cmdb-insight/docs/NORMALIZED-DATABASE-IMPLEMENTATION-PLAN.md

# Normalized Database Implementation Plan

## Executive Summary

**Status:** Green Field Implementation  
**Approach:** Complete rebuild with normalized structure  
**Timeline:** 1-2 weeks  
**Risk:** Low (no production data to migrate)

This document outlines the complete implementation plan for migrating from JSONB-based storage to a fully normalized, generic database structure that works with any Jira Assets schema.

---

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Database Schema](#database-schema)
3. [Implementation Components](#implementation-components)
4. [Implementation Steps](#implementation-steps)
5. [Code Structure](#code-structure)
6. [Testing Strategy](#testing-strategy)
7. [Migration Path](#migration-path)
8. [Example Implementations](#example-implementations)

---

## Architecture Overview

### Current Architecture (JSONB)

```
Jira Assets API
    ↓
jiraAssetsClient.parseObject()
    ↓
CMDBObject (TypeScript)
    ↓
cacheStore.upsertObject()
    ↓
cached_objects.data (JSONB column)
    ↓
Queries: Load all → Filter in JavaScript ❌
```

**Problems:**
- All objects loaded into memory
- No database-level indexing on attributes
- Slow queries with complex filters
- Memory overhead for large datasets

### Target Architecture (Normalized)

```
Jira Assets API
    ↓
jiraAssetsClient.parseObject()
    ↓
CMDBObject (TypeScript)
    ↓
normalizedCacheStore.upsertObject()
    ↓
Normalize to: objects + attribute_values
    ↓
Queries: SQL JOINs with indexes ✅
```

**Benefits:**
- Database-level filtering
- Indexed attributes
- Efficient queries
- Scalable to large datasets

### Key Benefits

1. **Generic**: Works with any Jira Assets schema (discovered dynamically)
2. **Efficient**: Database-level filtering with indexes
3. **Scalable**: Handles large datasets efficiently
4. **Type-safe**: Proper data types per attribute
5. **Queryable**: Complex filters at database level

---

## Database Schema

### Tables

#### 1. `object_types`
Stores discovered object types from Jira schema.

```sql
CREATE TABLE object_types (
  jira_type_id INTEGER PRIMARY KEY,
  type_name TEXT NOT NULL UNIQUE,
  display_name TEXT NOT NULL,
  description TEXT,
  sync_priority INTEGER DEFAULT 0,
  object_count INTEGER DEFAULT 0,
  discovered_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
```

**Purpose:** Metadata about object types discovered from Jira schema.

#### 2. `attributes`
Stores discovered attributes per object type.

```sql
CREATE TABLE attributes (
  id SERIAL PRIMARY KEY,
  jira_attr_id INTEGER NOT NULL,
  object_type_name TEXT NOT NULL REFERENCES object_types(type_name) ON DELETE CASCADE,
  attr_name TEXT NOT NULL,           -- "Application Function"
  field_name TEXT NOT NULL,          -- "applicationFunction" (camelCase)
  attr_type TEXT NOT NULL,           -- 'text', 'reference', 'integer', etc.
  is_multiple BOOLEAN NOT NULL DEFAULT FALSE,
  is_editable BOOLEAN NOT NULL DEFAULT TRUE,
  is_required BOOLEAN NOT NULL DEFAULT FALSE,
  is_system BOOLEAN NOT NULL DEFAULT FALSE,
  reference_type_name TEXT,          -- For reference attributes (e.g., "ApplicationFunction")
  description TEXT,
  discovered_at TIMESTAMP NOT NULL DEFAULT NOW(),
  UNIQUE(jira_attr_id, object_type_name)
);
```

**Purpose:** Metadata about attributes discovered from Jira schema. Used to build queries dynamically.

#### 3. `objects`
Stores minimal object metadata.

```sql
CREATE TABLE objects (
  id TEXT PRIMARY KEY,
  object_key TEXT NOT NULL UNIQUE,
  object_type_name TEXT NOT NULL REFERENCES object_types(type_name) ON DELETE CASCADE,
  label TEXT NOT NULL,
  jira_updated_at TIMESTAMP,
  jira_created_at TIMESTAMP,
  cached_at TIMESTAMP NOT NULL DEFAULT NOW()
);
```

**Purpose:** Core object information. All attribute values stored separately in `attribute_values`.

#### 4. `attribute_values`
EAV pattern for storing all attribute values.

```sql
CREATE TABLE attribute_values (
  id SERIAL PRIMARY KEY,
  object_id TEXT NOT NULL REFERENCES objects(id) ON DELETE CASCADE,
  attribute_id INTEGER NOT NULL REFERENCES attributes(id) ON DELETE CASCADE,
  -- Value storage (one column populated based on type):
  text_value TEXT,                   -- For text, textarea, url, email, select, user, status
  number_value NUMERIC,              -- For integer, float
  boolean_value BOOLEAN,             -- For boolean
  date_value DATE,                   -- For date
  datetime_value TIMESTAMP,          -- For datetime
  reference_object_id TEXT,          -- For reference attributes (stores objectId)
  -- For arrays: multiple rows with different array_index
  array_index INTEGER DEFAULT 0,     -- 0 = single value, >0 = array element
  UNIQUE(object_id, attribute_id, array_index)
);
```

**Purpose:** Stores all attribute values in normalized form. One row per value (multiple rows for arrays).

#### 5. `object_relations`
Enhanced existing table with attribute_id reference.

```sql
CREATE TABLE object_relations (
  id SERIAL PRIMARY KEY,
  source_id TEXT NOT NULL REFERENCES objects(id) ON DELETE CASCADE,
  target_id TEXT NOT NULL REFERENCES objects(id) ON DELETE CASCADE,
  attribute_id INTEGER NOT NULL REFERENCES attributes(id) ON DELETE CASCADE,
  source_type TEXT NOT NULL,
  target_type TEXT NOT NULL,
  UNIQUE(source_id, target_id, attribute_id)
);
```

**Purpose:** Stores relationships between objects. Enhanced with `attribute_id` for better queries.

#### 6. `sync_metadata`
Unchanged.

```sql
CREATE TABLE sync_metadata (
  key TEXT PRIMARY KEY,
  value TEXT NOT NULL,
  updated_at TEXT NOT NULL
);
```

### Indexes

**Critical for query performance:**

```sql
-- Objects
CREATE INDEX idx_objects_type ON objects(object_type_name);
CREATE INDEX idx_objects_key ON objects(object_key);
CREATE INDEX idx_objects_label ON objects(label);

-- Attributes (for schema lookups)
CREATE INDEX idx_attributes_type ON attributes(object_type_name);
CREATE INDEX idx_attributes_field ON attributes(field_name);
CREATE INDEX idx_attributes_type_field ON attributes(object_type_name, field_name);

-- Attribute Values (critical for query performance)
CREATE INDEX idx_attr_values_object ON attribute_values(object_id);
CREATE INDEX idx_attr_values_attr ON attribute_values(attribute_id);
CREATE INDEX idx_attr_values_text ON attribute_values(text_value) WHERE text_value IS NOT NULL;
CREATE INDEX idx_attr_values_number ON attribute_values(number_value) WHERE number_value IS NOT NULL;
CREATE INDEX idx_attr_values_reference ON attribute_values(reference_object_id) WHERE reference_object_id IS NOT NULL;
CREATE INDEX idx_attr_values_composite_text ON attribute_values(attribute_id, text_value) WHERE text_value IS NOT NULL;
CREATE INDEX idx_attr_values_composite_ref ON attribute_values(attribute_id, reference_object_id) WHERE reference_object_id IS NOT NULL;
CREATE INDEX idx_attr_values_object_attr ON attribute_values(object_id, attribute_id);

-- Relations
CREATE INDEX idx_relations_source ON object_relations(source_id);
CREATE INDEX idx_relations_target ON object_relations(target_id);
CREATE INDEX idx_relations_attr ON object_relations(attribute_id);
```

---

## Implementation Components

### 1. Schema Discovery Service

**File:** `backend/src/services/schemaDiscoveryService.ts`

**Purpose:** Populate `object_types` and `attributes` tables from generated schema.

**Methods:**

```typescript
class SchemaDiscoveryService {
  /**
   * Discover schema from OBJECT_TYPES and populate database
   */
  async discoverAndStoreSchema(): Promise<void>;
  
  /**
   * Get object type definition from database
   */
  async getObjectType(typeName: string): Promise<ObjectTypeDefinition | null>;
  
  /**
   * Get attribute definition by type and field name
   */
  async getAttribute(typeName: string, fieldName: string): Promise<AttributeDefinition | null>;
  
  /**
   * Get all attributes for an object type
   */
  async getAttributesForType(typeName: string): Promise<AttributeDefinition[]>;
}
```

**Implementation Notes:**
- Reads from `OBJECT_TYPES` in `generated/jira-schema.ts`
- Populates `object_types` and `attributes` tables
- Called once at startup or when schema changes
- Idempotent (can be called multiple times)

### 2. Normalized Cache Store

**File:** `backend/src/services/normalizedCacheStore.ts`

**Purpose:** Replace `cacheStore.ts` with normalized implementation.

**Key Methods:**

#### Object Operations
```typescript
class NormalizedCacheStore {
  /**
   * Upsert object: normalize and store
   */
  async upsertObject<T extends CMDBObject>(
    typeName: CMDBObjectTypeName,
    object: T
  ): Promise<void>;
  
  /**
   * Get object: reconstruct from normalized data
   */
  async getObject<T extends CMDBObject>(
    typeName: CMDBObjectTypeName,
    id: string
  ): Promise<T | null>;
  
  /**
   * Get objects with optional filters
   */
  async getObjects<T extends CMDBObject>(
    typeName: CMDBObjectTypeName,
    options?: QueryOptions
  ): Promise<T[]>;
  
  /**
   * Delete object and all its attribute values
   */
  async deleteObject(
    typeName: CMDBObjectTypeName,
    id: string
  ): Promise<boolean>;
}
```

#### Normalization Logic
```typescript
  /**
   * Convert CMDBObject to normalized form
   */
  private async normalizeObject(
    object: CMDBObject,
    typeName: CMDBObjectTypeName
  ): Promise<void>;
  
  /**
   * Store attribute value based on type
   */
  private async storeAttributeValue(
    objectId: string,
    attributeId: number,
    value: unknown,
    attrDef: AttributeDefinition,
    arrayIndex?: number
  ): Promise<void>;
```

#### Reconstruction Logic
```typescript
  /**
   * Reconstruct CMDBObject from normalized data
   */
  private async reconstructObject<T extends CMDBObject>(
    objectId: string,
    typeName: CMDBObjectTypeName
  ): Promise<T | null>;
  
  /**
   * Load all attribute values for an object
   */
  private async loadAttributeValues(
    objectId: string,
    typeName: CMDBObjectTypeName
  ): Promise<Map<string, unknown>>;
```

#### Query Operations
```typescript
  /**
   * Query objects with filters (generic)
   */
  async queryWithFilters<T extends CMDBObject>(
    typeName: CMDBObjectTypeName,
    filters: Record<string, unknown>,
    options?: QueryOptions
  ): Promise<{ objects: T[]; total: number }>;
```

### 3. Generic Query Builder

**File:** `backend/src/services/queryBuilder.ts`

**Purpose:** Build SQL queries dynamically based on filters.

**Methods:**

```typescript
class QueryBuilder {
  /**
   * Build WHERE clause from filters
   */
  buildWhereClause(
    filters: Record<string, unknown>,
    typeName: CMDBObjectTypeName
  ): { whereClause: string; params: unknown[] };
  
  /**
   * Build filter condition for one field
   */
  buildFilterCondition(
    fieldName: string,
    filterValue: unknown,
    attrDef: AttributeDefinition
  ): { condition: string; params: unknown[] };
  
  /**
   * Build ORDER BY clause
   */
  buildOrderBy(orderBy?: string, orderDir?: 'ASC' | 'DESC'): string;
  
  /**
   * Build pagination clause
   */
  buildPagination(limit?: number, offset?: number): string;
}
```

**Filter Types Supported:**

1. **Exact match:**
   ```typescript
   { status: "Active" }
   ```

2. **Contains (text):**
   ```typescript
   { name: { contains: "search" } }
   ```

3. **Reference match:**
   ```typescript
   { governanceModel: { objectId: "123" } }
   { governanceModel: { objectKey: "GOV-A" } }
   { governanceModel: { label: "Model A" } }
   ```

4. **Array contains:**
   ```typescript
   { applicationFunction: [
     { objectId: "1" },
     { objectId: "2" }
   ]}
   ```

5. **Exists:**
   ```typescript
   { applicationFunction: { exists: true } }
   ```

6. **Empty:**
   ```typescript
   { applicationFunction: { empty: true } }
   ```

### 4. Updated Services

#### syncEngine.ts
- Update `syncObjectType()` to use `normalizedCacheStore`
- Keep same interface, different implementation
- No changes to sync logic, only storage layer

#### dataService.ts
- Update `searchApplications()` to use `queryWithFilters()`
- Remove JavaScript filtering logic
- Use SQL queries instead
- Much faster for complex filters

#### cmdbService.ts
- Update to use `normalizedCacheStore` instead of `cacheStore`
- Keep same interface
- No changes to business logic

---

## Implementation Steps

### Step 1: Create Database Schema

**Files:**
- `backend/src/services/database/normalized-schema.ts` - Schema definitions
- Update `backend/src/services/database/migrations.ts` - Add migration

**Tasks:**
1. Create schema SQL for PostgreSQL
2. Create schema SQL for SQLite (development)
3. Create migration function
4. Test schema creation

**Deliverables:**
- Schema SQL files
- Migration function
- Tests pass

### Step 2: Schema Discovery Service

**File:** `backend/src/services/schemaDiscoveryService.ts`

**Tasks:**
1. Implement `discoverAndStoreSchema()` - Read from `OBJECT_TYPES`
2. Populate `object_types` table
3. Populate `attributes` table
4. Add validation and error handling
5. Add caching (don't rediscover if already done)

**Deliverables:**
- Schema discovery service
- Tests pass
- Schema populated in database

### Step 3: Normalization Logic

**File:** `backend/src/services/normalizedCacheStore.ts` (partial)

**Tasks:**
1. Implement `normalizeObject()` - Convert CMDBObject to normalized form
2. Handle all attribute types:
   - Text (text, textarea, url, email, select, user, status)
   - Numbers (integer, float)
   - Boolean
   - Dates (date, datetime)
   - References (single and multiple)
3. Implement `storeAttributeValue()` - Store values in correct columns
4. Handle arrays (multiple rows with array_index)
5. Test normalization with sample objects

**Deliverables:**
- Normalization logic
- Tests for all attribute types
- Sample data normalized correctly

### Step 4: Reconstruction Logic

**File:** `backend/src/services/normalizedCacheStore.ts` (partial)

**Tasks:**
1. Implement `reconstructObject()` - Build CMDBObject from normalized data
2. Load all attribute values for object
3. Convert back to TypeScript types
4. Handle arrays (multiple rows)
5. Handle references (load referenced objects if needed)
6. Test reconstruction correctness

**Deliverables:**
- Reconstruction logic
- Tests for all attribute types
- Reconstructed objects match original

### Step 5: Basic CRUD Operations

**File:** `backend/src/services/normalizedCacheStore.ts` (partial)

**Tasks:**
1. Implement `upsertObject()` - Normalize and store
2. Implement `getObject()` - Reconstruct and return
3. Implement `getObjects()` - Basic query (no filters yet)
4. Implement `deleteObject()` - Delete object and values
5. Implement `countObjects()` - Count by type
6. Test CRUD operations

**Deliverables:**
- CRUD operations working
- Tests pass
- Data integrity verified

### Step 6: Generic Query Builder

**File:** `backend/src/services/queryBuilder.ts`

**Tasks:**
1. Implement filter condition builder
2. Support all filter types:
   - Exact match
   - Contains
   - Reference match
   - Array contains
   - Exists
   - Empty
3. Build WHERE clauses dynamically
4. Handle JOINs for attribute values
5. Test query generation

**Deliverables:**
- Query builder
- Tests for all filter types
- SQL generation correct

### Step 7: Query Operations

**File:** `backend/src/services/normalizedCacheStore.ts` (partial)

**Tasks:**
1. Implement `queryWithFilters()` - Use query builder
2. Implement `countWithFilters()` - Count with same filters
3. Add pagination support
4. Add sorting support
5. Test complex queries

**Deliverables:**
- Query operations working
- Performance tests pass
- Complex filters work correctly

### Step 8: Relations

**File:** `backend/src/services/normalizedCacheStore.ts` (partial)

**Tasks:**
1. Update `extractAndStoreRelations()` - Use attribute_id
2. Update `getRelatedObjects()` - Use normalized queries
3. Update `getReferencingObjects()` - Use normalized queries
4. Test relations

**Deliverables:**
- Relations working
- Tests pass
- Performance acceptable

### Step 9: Update Services

**Files:**
- `backend/src/services/syncEngine.ts`
- `backend/src/services/cmdbService.ts`
- `backend/src/services/dataService.ts`

**Tasks:**
1. Replace `cacheStore` imports with `normalizedCacheStore`
2. Update `dataService.searchApplications()` - Use `queryWithFilters()`
3. Remove JavaScript filtering logic
4. Update all service calls
5. Test all endpoints

**Deliverables:**
- All services updated
- All endpoints working
- Tests pass

### Step 10: Statistics & Utilities

**File:** `backend/src/services/normalizedCacheStore.ts` (partial)

**Tasks:**
1. Implement `getStats()` - Count from normalized tables
2. Implement `isWarm()` - Check if cache has data
3. Implement `clearObjectType()` - Clear type and values
4. Implement `clearAll()` - Clear all data
5. Test statistics

**Deliverables:**
- Statistics working
- Tests pass
- Performance acceptable

### Step 11: Remove Old Code

**Files:**
- `backend/src/services/cacheStore.ts` - Delete or archive

**Tasks:**
1. Remove old `cacheStore.ts`
2. Update all imports
3. Clean up unused code
4. Update documentation

**Deliverables:**
- Old code removed
- All imports updated
- Documentation updated

---

## Code Structure

### File Organization

```
backend/src/services/
├── database/
│   ├── normalized-schema.ts          # NEW: Schema definitions
│   ├── migrations.ts                 # UPDATED: Add normalized schema migration
│   ├── factory.ts                    # (unchanged)
│   ├── interface.ts                  # (unchanged)
│   ├── postgresAdapter.ts            # (unchanged)
│   └── sqliteAdapter.ts              # (unchanged)
├── schemaDiscoveryService.ts          # NEW: Schema discovery
├── normalizedCacheStore.ts           # NEW: Normalized storage
├── queryBuilder.ts                   # NEW: Generic query builder
├── syncEngine.ts                     # UPDATED: Use normalized store
├── cmdbService.ts                    # UPDATED: Use normalized store
├── dataService.ts                    # UPDATED: Use queries
└── cacheStore.ts                     # REMOVE: Old implementation
```

### Key Interfaces

```typescript
// NormalizedCacheStore interface (same as old CacheStore)
interface NormalizedCacheStore {
  // Object operations
  upsertObject<T>(typeName: CMDBObjectTypeName, object: T): Promise<void>;
  getObject<T>(typeName: CMDBObjectTypeName, id: string): Promise<T | null>;
  getObjectByKey<T>(typeName: CMDBObjectTypeName, objectKey: string): Promise<T | null>;
  getObjects<T>(typeName: CMDBObjectTypeName, options?: QueryOptions): Promise<T[]>;
  countObjects(typeName: CMDBObjectTypeName): Promise<number>;
  deleteObject(typeName: CMDBObjectTypeName, id: string): Promise<boolean>;
  
  // Query operations
  queryWithFilters<T>(
    typeName: CMDBObjectTypeName,
    filters: Record<string, unknown>,
    options?: QueryOptions
  ): Promise<{ objects: T[]; total: number }>;
  
  // Relations
  extractAndStoreRelations<T>(typeName: CMDBObjectTypeName, object: T): Promise<void>;
  getRelatedObjects<T>(
    sourceId: string,
    targetTypeName: CMDBObjectTypeName,
    attributeName?: string
  ): Promise<T[]>;
  getReferencingObjects<T>(
    targetId: string,
    sourceTypeName: CMDBObjectTypeName,
    attributeName?: string
  ): Promise<T[]>;
  
  // Statistics
  getStats(): Promise<CacheStats>;
  isWarm(): Promise<boolean>;
  clearObjectType(typeName: CMDBObjectTypeName): Promise<number>;
  clearAll(): Promise<void>;
  
  // Sync metadata
  getSyncMetadata(key: string): Promise<string | null>;
  setSyncMetadata(key: string, value: string): Promise<void>;
}

// QueryBuilder interface
interface QueryBuilder {
  buildWhereClause(
    filters: Record<string, unknown>,
    typeName: CMDBObjectTypeName
  ): { whereClause: string; params: unknown[] };
  
  buildFilterCondition(
    fieldName: string,
    filterValue: unknown,
    attrDef: AttributeDefinition
  ): { condition: string; params: unknown[] };
}
```

---

## Testing Strategy

### Unit Tests

1. **Schema Discovery**
   - Test schema population
   - Test attribute lookup
   - Test object type lookup
   - Test idempotency

2. **Normalization**
   - Test all attribute types (text, number, boolean, date, reference)
   - Test arrays (multiple values)
   - Test null/empty values
   - Test edge cases

3. **Reconstruction**
   - Test object reconstruction
   - Test all attribute types
   - Test arrays
   - Test references
   - Test missing values

4. **Query Builder**
   - Test all filter types
   - Test complex filters
   - Test SQL generation
   - Test parameter binding

### Integration Tests

1. **CRUD Operations**
   - Create, read, update, delete
   - Verify data integrity
   - Test transactions

2. **Queries**
   - Simple filters
   - Complex filters (multiple conditions)
   - Pagination
   - Sorting
   - Performance with large datasets

3. **Relations**
   - Store relations
   - Query relations
   - Delete relations
   - Cascade deletes

### Performance Tests

1. **Query Performance**
   - Compare old vs new
   - Test with 500+ objects
   - Test complex filters
   - Measure query time

2. **Write Performance**
   - Batch inserts
   - Single inserts
   - Updates
   - Measure write time

3. **Memory Usage**
   - Compare old vs new
   - Test with large datasets
   - Measure memory footprint

---

## Migration Path

### Since it's Green Field

1. **No Data Migration Needed**
   - Start fresh with normalized structure
   - No existing data to migrate
   - Clean implementation

2. **Implementation Order**
   - Build new normalized structure
   - Test thoroughly
   - Replace old code
   - Deploy

3. **Rollback Plan**
   - Keep old code in git history
   - Can revert if needed
   - No data loss risk (green field)

---

## Example Implementations

### Example 1: Normalization

**Input (CMDBObject):**
```typescript
{
  id: "123",
  objectKey: "APP-1",
  label: "My Application",
  status: "Active",
  applicationFunction: [
    { objectId: "456", objectKey: "FUNC-1", label: "Function 1" },
    { objectId: "789", objectKey: "FUNC-2", label: "Function 2" }
  ],
  ictGovernanceModel: { objectId: "999", objectKey: "GOV-A", label: "Model A" },
  customDevelopment: true,
  zenyaID: 42
}
```

**Normalized (objects + attribute_values):**

```sql
-- objects table
INSERT INTO objects (id, object_key, object_type_name, label, jira_updated_at, jira_created_at, cached_at)
VALUES ('123', 'APP-1', 'ApplicationComponent', 'My Application', '2024-01-01', '2024-01-01', NOW());

-- attribute_values table
INSERT INTO attribute_values (object_id, attribute_id, text_value, number_value, boolean_value, reference_object_id, array_index)
VALUES 
  -- status (text)
  ('123', <status_attr_id>, 'Active', NULL, NULL, NULL, 0),
  -- applicationFunction (reference array)
  ('123', <appfunc_attr_id>, NULL, NULL, NULL, '456', 0),  -- First function
  ('123', <appfunc_attr_id>, NULL, NULL, NULL, '789', 1),  -- Second function
  -- ictGovernanceModel (reference)
  ('123', <gov_attr_id>, NULL, NULL, NULL, '999', 0),
  -- customDevelopment (boolean)
  ('123', <customdev_attr_id>, NULL, NULL, true, NULL, 0),
  -- zenyaID (integer)
  ('123', <zenyaid_attr_id>, NULL, 42, NULL, NULL, 0);
```

### Example 2: Query

**Filter:**
```typescript
{
  status: "Active",
  governanceModel: { objectId: "999" },
  applicationFunction: { exists: true }
}
```

**Generated SQL:**
```sql
SELECT DISTINCT o.* 
FROM objects o
WHERE o.object_type_name = 'ApplicationComponent'
  AND EXISTS (
    SELECT 1 FROM attribute_values av
    JOIN attributes a ON av.attribute_id = a.id
    WHERE av.object_id = o.id
      AND a.field_name = 'status'
      AND av.text_value = $1
  )
  AND EXISTS (
    SELECT 1 FROM attribute_values av
    JOIN attributes a ON av.attribute_id = a.id
    WHERE av.object_id = o.id
      AND a.field_name = 'ictGovernanceModel'
      AND av.reference_object_id = $2
  )
  AND EXISTS (
    SELECT 1 FROM attribute_values av
    JOIN attributes a ON av.attribute_id = a.id
    WHERE av.object_id = o.id
      AND a.field_name = 'applicationFunction'
  )
ORDER BY o.label ASC
LIMIT $3 OFFSET $4;
```

**Parameters:** `['Active', '999', 25, 0]`

### Example 3: Reconstruction

**Query:**
```sql
SELECT o.*, a.field_name, av.*
FROM objects o
LEFT JOIN attribute_values av ON av.object_id = o.id
LEFT JOIN attributes a ON av.attribute_id = a.id
WHERE o.id = '123'
ORDER BY a.field_name, av.array_index;
```

**Result Processing:**
```typescript
// Group by field_name
const values: Record<string, unknown> = {};

for (const row of rows) {
  const fieldName = row.field_name;
  const value = getValueFromRow(row); // Extract from correct column
  
  if (row.array_index === 0) {
    // Single value
    values[fieldName] = value;
  } else {
    // Array value
    if (!values[fieldName]) values[fieldName] = [];
    (values[fieldName] as unknown[]).push(value);
  }
}

// Build CMDBObject
const object: CMDBObject = {
  id: row.id,
  objectKey: row.object_key,
  label: row.label,
  ...values
};
```

---

## Success Criteria

### Functional Requirements

- ✅ All existing functionality works
- ✅ Queries return correct results
- ✅ No data loss
- ✅ Relations work correctly
- ✅ Sync process works

### Performance Requirements

- ✅ Query performance: 50%+ faster for filtered queries
- ✅ Memory usage: 30%+ reduction
- ✅ Write performance: No degradation (< 10% slower acceptable)
- ✅ Database size: Similar or smaller

### Quality Requirements

- ✅ Test coverage: 80%+ for new code
- ✅ No critical bugs
- ✅ Code well documented
- ✅ Type-safe implementation

---

## Timeline

| Step | Duration | Description |
|------|----------|-------------|
| 1 | 0.5 day | Database schema |
| 2 | 0.5 day | Schema discovery |
| 3 | 1 day | Normalization logic |
| 4 | 1 day | Reconstruction logic |
| 5 | 0.5 day | Basic CRUD |
| 6 | 1 day | Query builder |
| 7 | 1 day | Query operations |
| 8 | 0.5 day | Relations |
| 9 | 1 day | Update services |
| 10 | 0.5 day | Statistics |
| 11 | 0.5 day | Cleanup |
| **Total** | **8 days** | |

**Buffer:** +2 days for unexpected issues  
**Total Estimated:** 1.5-2 weeks

---

## Next Steps

1. ✅ Review and approve plan
2. ✅ Create feature branch: `feature/normalized-database`
3. ✅ Start implementation (Step 1)
4. ✅ Daily progress updates
5. ✅ Weekly review

---

## Appendix

### A. Attribute Type Mapping

| Jira Type | Database Column | TypeScript Type |
|-----------|----------------|-----------------|
| text, textarea, url, email, select, user, status | text_value | string |
| integer, float | number_value | number |
| boolean | boolean_value | boolean |
| date | date_value | string (ISO date) |
| datetime | datetime_value | string (ISO datetime) |
| reference | reference_object_id | ObjectReference |

### B. Query Performance Comparison

**Old (JSONB + JavaScript filtering):**
- Load all objects: ~500ms
- Filter in JavaScript: ~50ms
- Total: ~550ms

**New (Normalized + SQL):**
- Query with indexes: ~20ms
- Total: ~20ms

**Improvement:** ~27x faster

### C. Database Size Comparison

**Old (JSONB):**
- 500 objects × ~5KB JSON = ~2.5MB

**New (Normalized):**
- 500 objects × ~100 bytes = ~50KB (objects)
- 500 objects × ~20 attributes × ~50 bytes = ~500KB (attribute_values)
- Total: ~550KB

**Improvement:** ~4.5x smaller

---

**End of Plan**