Bert Hausmans
cdee0e8819
- Restore blue PageHeader on Dashboard (/app-components) - Update homepage (/) with subtle header design without blue bar - Add uniform PageHeader styling to application edit page - Fix Rapporten link on homepage to point to /reports overview - Improve header descriptions spacing for better readability
802 lines
30 KiB
Markdown
802 lines
30 KiB
Markdown
# Refactor Plan - Phase 1: Architecture Analysis
|
|
|
|
**Created:** 2025-01-XX
|
|
**Status:** Phase 1 - Analysis Only (No functional changes)
|
|
|
|
## Executive Summary
|
|
|
|
This document provides a comprehensive analysis of the current architecture and a plan for refactoring the CMDB Insight codebase to improve maintainability, reduce duplication, and establish clearer separation of concerns.
|
|
|
|
**Scope:** This is Phase 1 - analysis and planning only. No code changes will be made in this phase.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Current Architecture Map](#current-architecture-map)
|
|
2. [Pain Points & Duplication](#pain-points--duplication)
|
|
3. [Target Architecture](#target-architecture)
|
|
4. [Migration Steps](#migration-steps)
|
|
5. [Explicit Deletion List](#explicit-deletion-list)
|
|
6. [API Payload Contract & Recursion Insights](#api-payload-contract--recursion-insights)
|
|
|
|
---
|
|
|
|
## Current Architecture Map
|
|
|
|
### File/Folder Structure
|
|
|
|
```
|
|
backend/src/
|
|
├── services/
|
|
│ ├── jiraAssets.ts # High-level Jira Assets service (business logic, ~3454 lines)
|
|
│ ├── jiraAssetsClient.ts # Low-level Jira Assets API client (~646 lines)
|
|
│ ├── schemaDiscoveryService.ts # Discovers schema from Jira API (~520 lines)
|
|
│ ├── schemaCacheService.ts # Caches schema metadata
|
|
│ ├── schemaConfigurationService.ts # Manages enabled object types
|
|
│ ├── schemaMappingService.ts # Maps object types to schema IDs
|
|
│ ├── syncEngine.ts # Background sync service (full/incremental) (~630 lines)
|
|
│ ├── normalizedCacheStore.ts # EAV pattern DB store (~1695 lines)
|
|
│ ├── cmdbService.ts # Universal schema-driven CMDB service (~531 lines)
|
|
│ ├── queryBuilder.ts # Dynamic SQL query builder (~278 lines)
|
|
│ ├── cacheStore.old.ts # Legacy cache store (deprecated)
|
|
│ └── database/
|
|
│ ├── normalized-schema.ts # DB schema definitions (Postgres/SQLite)
|
|
│ ├── factory.ts # Database adapter factory
|
|
│ ├── interface.ts # Database adapter interface
|
|
│ ├── postgresAdapter.ts # PostgreSQL adapter
|
|
│ ├── sqliteAdapter.ts # SQLite adapter
|
|
│ ├── migrate-to-normalized-schema.ts
|
|
│ └── fix-object-types-constraints.ts
|
|
├── routes/
|
|
│ ├── applications.ts # Application-specific endpoints (~780 lines)
|
|
│ ├── objects.ts # Generic object endpoints (~185 lines)
|
|
│ ├── cache.ts # Cache/sync endpoints (~165 lines)
|
|
│ ├── schema.ts # Schema endpoints (~107 lines)
|
|
│ └── schemaConfiguration.ts # Schema configuration endpoints
|
|
├── generated/
|
|
│ ├── jira-types.ts # Generated TypeScript types (~934 lines)
|
|
│ └── jira-schema.ts # Generated schema metadata (~895 lines)
|
|
└── scripts/
|
|
├── discover-schema.ts # Schema discovery CLI
|
|
├── generate-types-from-db.ts # Type generation from DB (~485 lines)
|
|
└── generate-schema.ts # Legacy schema generation
|
|
```
|
|
|
|
### Module Responsibilities
|
|
|
|
#### 1. Jira Assets API Client Calls
|
|
|
|
**Primary Files:**
|
|
- `services/jiraAssetsClient.ts` - Low-level HTTP client
|
|
- Methods: `getObject()`, `searchObjects()`, `getAllObjectsOfType()`, `updateObject()`, `parseObject()`
|
|
- Handles authentication (service account token for reads, user PAT for writes)
|
|
- API detection (Data Center vs Cloud)
|
|
- Object parsing from Jira format to CMDB format
|
|
|
|
- `services/jiraAssets.ts` - High-level business logic wrapper
|
|
- Application-specific methods (e.g., `getApplications()`, `updateApplication()`)
|
|
- Dashboard data aggregation
|
|
- Reference data caching
|
|
- Team dashboard calculations
|
|
- Legacy API methods
|
|
|
|
**Dependencies:**
|
|
- Uses `schemaCacheService` for type lookups
|
|
- Uses `schemaMappingService` for schema ID resolution
|
|
|
|
#### 2. Schema Discovery/Sync
|
|
|
|
**Primary Files:**
|
|
- `services/schemaDiscoveryService.ts`
|
|
- Discovers object types from Jira API (`/objectschema/{id}/objecttypes/flat`)
|
|
- Discovers attributes for each object type (`/objecttype/{id}/attributes`)
|
|
- Stores schema in database (`object_types`, `attributes` tables)
|
|
- Provides lookup methods: `getAttribute()`, `getAttributesForType()`, `getObjectType()`
|
|
|
|
- `services/schemaCacheService.ts`
|
|
- Caches schema from database
|
|
- Provides runtime schema access
|
|
|
|
- `services/schemaConfigurationService.ts`
|
|
- Manages enabled/disabled object types
|
|
- Schema-to-object-type mapping
|
|
- Configuration validation
|
|
|
|
- `services/schemaMappingService.ts`
|
|
- Maps object type names to schema IDs
|
|
- Legacy compatibility
|
|
|
|
**Scripts:**
|
|
- `scripts/discover-schema.ts` - CLI tool to trigger schema discovery
|
|
- `scripts/generate-types-from-db.ts` - Generates TypeScript types from database
|
|
|
|
#### 3. Object Sync/Import
|
|
|
|
**Primary Files:**
|
|
- `services/syncEngine.ts`
|
|
- `fullSync()` - Syncs all enabled object types
|
|
- `incrementalSync()` - Periodic sync of updated objects
|
|
- `syncType()` - Sync single object type
|
|
- `syncObject()` - Sync single object
|
|
- Uses `jiraAssetsClient.getAllObjectsOfType()` for fetching
|
|
- Uses `normalizedCacheStore.batchUpsertObjects()` for storage
|
|
|
|
**Flow:**
|
|
1. Fetch objects from Jira via `jiraAssetsClient`
|
|
2. Parse objects via `jiraAssetsClient.parseObject()`
|
|
3. Store objects via `normalizedCacheStore.batchUpsertObjects()`
|
|
4. Extract relations via `normalizedCacheStore.extractAndStoreRelations()`
|
|
|
|
#### 4. DB Normalization Store (EAV)
|
|
|
|
**Primary Files:**
|
|
- `services/normalizedCacheStore.ts` (~1695 lines)
|
|
- **Storage:** `normalizeObject()`, `batchUpsertObjects()`, `upsertObject()`
|
|
- **Retrieval:** `getObject()`, `getObjects()`, `reconstructObject()`, `loadAttributeValues()`
|
|
- **Relations:** `extractAndStoreRelations()`, `getRelatedObjects()`, `getReferencingObjects()`
|
|
- **Query:** `queryWithFilters()` (uses `queryBuilder`)
|
|
|
|
- `services/database/normalized-schema.ts`
|
|
- Defines EAV schema: `objects`, `attributes`, `attribute_values`, `object_relations`
|
|
|
|
**EAV Pattern:**
|
|
- `objects` table: Minimal metadata (id, objectKey, label, type, timestamps)
|
|
- `attributes` table: Schema metadata (jira_attr_id, field_name, type, is_multiple, etc.)
|
|
- `attribute_values` table: Actual values (text_value, number_value, boolean_value, reference_object_id, array_index)
|
|
- `object_relations` table: Extracted relationships (source_id, target_id, attribute_id)
|
|
|
|
#### 5. Backend API Endpoints
|
|
|
|
**Primary Files:**
|
|
- `routes/applications.ts` - Application-specific endpoints
|
|
- `POST /applications/search` - Search with filters
|
|
- `GET /applications/:id` - Get application details
|
|
- `PUT /applications/:id` - Update application
|
|
- `GET /applications/:id/related/:type` - Get related objects
|
|
- Dashboard endpoints (`/team-dashboard`, `/team-portfolio-health`)
|
|
|
|
- `routes/objects.ts` - Generic object endpoints
|
|
- `GET /objects` - List supported types
|
|
- `GET /objects/:type` - Get all objects of type
|
|
- `GET /objects/:type/:id` - Get single object
|
|
- `GET /objects/:type/:id/related/:relationType` - Get related objects
|
|
|
|
- `routes/cache.ts` - Cache management
|
|
- `POST /cache/sync` - Trigger full sync
|
|
- `POST /cache/sync/:objectType` - Sync single type
|
|
- `POST /cache/refresh-application/:id` - Refresh single object
|
|
|
|
- `routes/schema.ts` - Schema endpoints
|
|
- `GET /schema` - Get schema metadata
|
|
- `GET /schema/types` - List object types
|
|
- `GET /schema/types/:type` - Get type definition
|
|
|
|
**Service Layer:**
|
|
- Routes delegate to `cmdbService`, `dataService`, `syncEngine`
|
|
- `cmdbService` provides unified interface (read/write with conflict detection)
|
|
- `dataService` provides application-specific business logic
|
|
|
|
#### 6. Query Builder (Object Reconstruction)
|
|
|
|
**Primary Files:**
|
|
- `services/queryBuilder.ts`
|
|
- `buildWhereClause()` - Builds WHERE conditions from filters
|
|
- `buildFilterCondition()` - Handles different attribute types (text, reference, number, etc.)
|
|
- `buildOrderBy()` - ORDER BY clause
|
|
- `buildPagination()` - LIMIT/OFFSET clause
|
|
|
|
**Usage:**
|
|
- Used by `normalizedCacheStore.queryWithFilters()` to build dynamic SQL
|
|
- Handles complex filters (exact match, exists, contains, reference filters)
|
|
|
|
#### 7. Generated Types/Reflect Scripts
|
|
|
|
**Primary Files:**
|
|
- `scripts/generate-types-from-db.ts`
|
|
- Reads from `object_types` and `attributes` tables
|
|
- Generates `generated/jira-types.ts` (TypeScript interfaces)
|
|
- Generates `generated/jira-schema.ts` (Schema metadata with lookup maps)
|
|
|
|
**Generated Output:**
|
|
- `jira-types.ts`: TypeScript interfaces for each object type (e.g., `ApplicationComponent`, `Server`)
|
|
- `jira-schema.ts`: `OBJECT_TYPES` record, lookup maps (`TYPE_ID_TO_NAME`, `JIRA_NAME_TO_TYPE`), helper functions
|
|
|
|
---
|
|
|
|
## Pain Points & Duplication
|
|
|
|
### 1. Dual API Clients (jiraAssets.ts vs jiraAssetsClient.ts)
|
|
|
|
**Issue:** Two separate services handling Jira API calls:
|
|
- `jiraAssetsClient.ts` - Low-level, focused on API communication
|
|
- `jiraAssets.ts` - High-level, contains business logic + API calls
|
|
|
|
**Problems:**
|
|
- Duplication of API request logic
|
|
- Inconsistent error handling
|
|
- Mixed concerns (business logic + infrastructure)
|
|
- `jiraAssets.ts` is huge (~3454 lines) and hard to maintain
|
|
|
|
**Location:**
|
|
- `backend/src/services/jiraAssets.ts` - Contains both API calls and business logic
|
|
- `backend/src/services/jiraAssetsClient.ts` - Clean separation but incomplete
|
|
|
|
### 2. Schema Discovery/Caching Duplication
|
|
|
|
**Issue:** Multiple services handling schema metadata:
|
|
- `schemaDiscoveryService.ts` - Discovers and stores schema
|
|
- `schemaCacheService.ts` - Caches schema from DB
|
|
- `schemaConfigurationService.ts` - Manages enabled types
|
|
- `schemaMappingService.ts` - Maps types to schema IDs
|
|
|
|
**Problems:**
|
|
- Unclear boundaries between services
|
|
- Potential for stale cache
|
|
- Complex initialization dependencies
|
|
|
|
**Location:**
|
|
- `backend/src/services/schema*.ts` files
|
|
|
|
### 3. Mixed Responsibilities in normalizedCacheStore.ts
|
|
|
|
**Issue:** Large file (~1695 lines) handling multiple concerns:
|
|
- Database operations (EAV storage/retrieval)
|
|
- Object reconstruction (TypeScript object building)
|
|
- Reference resolution (fetching missing referenced objects)
|
|
- Relation extraction
|
|
|
|
**Problems:**
|
|
- Hard to test individual concerns
|
|
- Difficult to optimize specific operations
|
|
- Violates single responsibility principle
|
|
|
|
**Location:**
|
|
- `backend/src/services/normalizedCacheStore.ts`
|
|
|
|
### 4. Application-Specific Logic in Generic Services
|
|
|
|
**Issue:** Application-specific business logic scattered:
|
|
- `routes/applications.ts` - Application-specific endpoints (~780 lines)
|
|
- `services/dataService.ts` - Application business logic
|
|
- `services/jiraAssets.ts` - Application aggregation logic
|
|
- `services/cmdbService.ts` - Generic service used by applications
|
|
|
|
**Problems:**
|
|
- Hard to extend to other object types
|
|
- Tight coupling between routes and services
|
|
- Business logic mixed with data access
|
|
|
|
**Location:**
|
|
- `backend/src/routes/applications.ts`
|
|
- `backend/src/services/dataService.ts`
|
|
|
|
### 5. Type Generation Pipeline Complexity
|
|
|
|
**Issue:** Multiple scripts and services involved in type generation:
|
|
- `scripts/discover-schema.ts` - Triggers schema discovery
|
|
- `services/schemaDiscoveryService.ts` - Discovers schema
|
|
- `scripts/generate-types-from-db.ts` - Generates TypeScript files
|
|
- `generated/jira-types.ts` - Generated output (must be regenerated when schema changes)
|
|
|
|
**Problems:**
|
|
- Unclear workflow
|
|
- Manual steps required
|
|
- Generated files can get out of sync
|
|
|
|
**Location:**
|
|
- `backend/scripts/discover-schema.ts`
|
|
- `backend/scripts/generate-types-from-db.ts`
|
|
- `backend/src/generated/*.ts`
|
|
|
|
### 6. Legacy Code (cacheStore.old.ts)
|
|
|
|
**Issue:** Old cache store still present in codebase:
|
|
- `services/cacheStore.old.ts` - Deprecated implementation
|
|
|
|
**Problems:**
|
|
- Confusing for new developers
|
|
- Takes up space
|
|
- No longer used
|
|
|
|
**Location:**
|
|
- `backend/src/services/cacheStore.old.ts`
|
|
|
|
### 7. Inconsistent Error Handling
|
|
|
|
**Issue:** Different error handling patterns across services:
|
|
- Some use try/catch with logger
|
|
- Some throw errors
|
|
- Some return null/undefined
|
|
- Inconsistent error messages
|
|
|
|
**Problems:**
|
|
- Hard to debug issues
|
|
- Inconsistent API responses
|
|
- No centralized error handling
|
|
|
|
**Location:**
|
|
- Throughout codebase
|
|
|
|
---
|
|
|
|
## Target Architecture
|
|
|
|
### Domain/Infrastructure/Services/API Separation
|
|
|
|
```
|
|
backend/src/
|
|
├── domain/ # Domain models & business logic
|
|
│ ├── cmdb/
|
|
│ │ ├── Object.ts # CMDBObject base interface
|
|
│ │ ├── ObjectType.ts # ObjectTypeDefinition
|
|
│ │ ├── Attribute.ts # AttributeDefinition
|
|
│ │ └── Reference.ts # ObjectReference
|
|
│ ├── schema/
|
|
│ │ ├── Schema.ts # Schema domain model
|
|
│ │ └── SchemaDiscovery.ts # Schema discovery business logic
|
|
│ └── sync/
|
|
│ ├── SyncEngine.ts # Sync orchestration logic
|
|
│ └── SyncStrategy.ts # Sync strategies (full, incremental)
|
|
│
|
|
├── infrastructure/ # External integrations & infrastructure
|
|
│ ├── jira/
|
|
│ │ ├── JiraAssetsClient.ts # Low-level HTTP client (pure API calls)
|
|
│ │ ├── JiraAssetsApi.ts # API contract definitions
|
|
│ │ └── JiraResponseParser.ts # Response parsing utilities
|
|
│ └── database/
|
|
│ ├── adapters/ # Database adapters (Postgres, SQLite)
|
|
│ ├── schema/ # Schema definitions
|
|
│ └── migrations/ # Database migrations
|
|
│
|
|
├── services/ # Application services (use cases)
|
|
│ ├── cmdb/
|
|
│ │ ├── CmdbReadService.ts # Read operations
|
|
│ │ ├── CmdbWriteService.ts # Write operations with conflict detection
|
|
│ │ └── CmdbQueryService.ts # Query operations
|
|
│ ├── schema/
|
|
│ │ ├── SchemaService.ts # Schema CRUD operations
|
|
│ │ └── SchemaDiscoveryService.ts # Schema discovery orchestration
|
|
│ └── sync/
|
|
│ └── SyncService.ts # Sync orchestration
|
|
│
|
|
├── repositories/ # Data access layer
|
|
│ ├── ObjectRepository.ts # Object CRUD (uses EAV store)
|
|
│ ├── AttributeRepository.ts # Attribute value access
|
|
│ ├── RelationRepository.ts # Relationship access
|
|
│ └── SchemaRepository.ts # Schema metadata access
|
|
│
|
|
├── stores/ # Storage implementations
|
|
│ ├── NormalizedObjectStore.ts # EAV pattern implementation
|
|
│ ├── ObjectReconstructor.ts # Object reconstruction from EAV
|
|
│ └── RelationExtractor.ts # Relation extraction logic
|
|
│
|
|
├── api/ # HTTP API layer
|
|
│ ├── routes/
|
|
│ │ ├── objects.ts # Generic object endpoints
|
|
│ │ ├── schema.ts # Schema endpoints
|
|
│ │ └── sync.ts # Sync endpoints
|
|
│ ├── handlers/ # Request handlers (thin layer)
|
|
│ │ ├── ObjectHandler.ts
|
|
│ │ ├── SchemaHandler.ts
|
|
│ │ └── SyncHandler.ts
|
|
│ └── middleware/ # Auth, validation, etc.
|
|
│
|
|
├── queries/ # Query builders
|
|
│ ├── ObjectQueryBuilder.ts # SQL query construction
|
|
│ └── FilterBuilder.ts # Filter condition builder
|
|
│
|
|
└── scripts/ # CLI tools
|
|
├── discover-schema.ts # Schema discovery CLI
|
|
└── generate-types.ts # Type generation CLI
|
|
```
|
|
|
|
### Key Principles
|
|
|
|
1. **Domain Layer**: Pure business logic, no infrastructure dependencies
|
|
2. **Infrastructure Layer**: External integrations (Jira API, database)
|
|
3. **Services Layer**: Orchestrates domain logic and infrastructure
|
|
4. **Repository Layer**: Data access abstraction
|
|
5. **Store Layer**: Storage implementations (EAV, caching)
|
|
6. **API Layer**: Thin HTTP handlers that delegate to services
|
|
|
|
---
|
|
|
|
## Migration Steps
|
|
|
|
### Step 1: Extract Jira API Client (Infrastructure)
|
|
|
|
**Goal:** Create pure infrastructure client with no business logic
|
|
|
|
1. Consolidate `jiraAssetsClient.ts` and `jiraAssets.ts` API methods into single `JiraAssetsClient`
|
|
2. Extract API contract types to `infrastructure/jira/JiraAssetsApi.ts`
|
|
3. Move response parsing to `infrastructure/jira/JiraResponseParser.ts`
|
|
4. Remove business logic from API client (delegate to services)
|
|
|
|
**Files to Create:**
|
|
- `infrastructure/jira/JiraAssetsClient.ts`
|
|
- `infrastructure/jira/JiraAssetsApi.ts`
|
|
- `infrastructure/jira/JiraResponseParser.ts`
|
|
|
|
**Files to Modify:**
|
|
- `services/jiraAssets.ts` - Remove API calls, keep business logic
|
|
- `services/jiraAssetsClient.ts` - Merge into infrastructure client
|
|
|
|
**Files to Delete:**
|
|
- None (yet - deprecate old files after migration)
|
|
|
|
### Step 2: Extract Schema Domain & Services
|
|
|
|
**Goal:** Separate schema discovery business logic from infrastructure
|
|
|
|
1. Create `domain/schema/` with domain models
|
|
2. Move schema discovery logic to `services/schema/SchemaDiscoveryService.ts`
|
|
3. Consolidate schema caching in `services/schema/SchemaService.ts`
|
|
4. Remove duplication between `schemaCacheService`, `schemaConfigurationService`, `schemaMappingService`
|
|
|
|
**Files to Create:**
|
|
- `domain/schema/Schema.ts`
|
|
- `services/schema/SchemaService.ts`
|
|
- `services/schema/SchemaDiscoveryService.ts`
|
|
|
|
**Files to Modify:**
|
|
- `services/schemaDiscoveryService.ts` - Split into domain + service
|
|
- `services/schemaCacheService.ts` - Merge into SchemaService
|
|
- `services/schemaConfigurationService.ts` - Merge into SchemaService
|
|
- `services/schemaMappingService.ts` - Merge into SchemaService
|
|
|
|
**Files to Delete:**
|
|
- `services/schemaCacheService.ts` (after merge)
|
|
- `services/schemaMappingService.ts` (after merge)
|
|
|
|
### Step 3: Extract Repository Layer
|
|
|
|
**Goal:** Abstract data access from business logic
|
|
|
|
1. Create `repositories/ObjectRepository.ts` - Interface for object CRUD
|
|
2. Create `repositories/AttributeRepository.ts` - Interface for attribute access
|
|
3. Create `repositories/RelationRepository.ts` - Interface for relationships
|
|
4. Implement repositories using `NormalizedObjectStore`
|
|
|
|
**Files to Create:**
|
|
- `repositories/ObjectRepository.ts`
|
|
- `repositories/AttributeRepository.ts`
|
|
- `repositories/RelationRepository.ts`
|
|
- `repositories/SchemaRepository.ts`
|
|
|
|
**Files to Modify:**
|
|
- `services/normalizedCacheStore.ts` - Extract repository implementations
|
|
|
|
### Step 4: Extract Store Implementations
|
|
|
|
**Goal:** Separate storage implementations from business logic
|
|
|
|
1. Extract EAV storage to `stores/NormalizedObjectStore.ts`
|
|
2. Extract object reconstruction to `stores/ObjectReconstructor.ts`
|
|
3. Extract relation extraction to `stores/RelationExtractor.ts`
|
|
|
|
**Files to Create:**
|
|
- `stores/NormalizedObjectStore.ts` - EAV storage/retrieval
|
|
- `stores/ObjectReconstructor.ts` - TypeScript object reconstruction
|
|
- `stores/RelationExtractor.ts` - Relation extraction from objects
|
|
|
|
**Files to Modify:**
|
|
- `services/normalizedCacheStore.ts` - Split into store classes
|
|
|
|
### Step 5: Extract Query Builders
|
|
|
|
**Goal:** Separate query construction from execution
|
|
|
|
1. Move `queryBuilder.ts` to `queries/ObjectQueryBuilder.ts`
|
|
2. Extract filter building to `queries/FilterBuilder.ts`
|
|
|
|
**Files to Create:**
|
|
- `queries/ObjectQueryBuilder.ts`
|
|
- `queries/FilterBuilder.ts`
|
|
|
|
**Files to Modify:**
|
|
- `services/queryBuilder.ts` - Move to queries/
|
|
|
|
### Step 6: Extract CMDB Services
|
|
|
|
**Goal:** Separate read/write/query concerns
|
|
|
|
1. Create `services/cmdb/CmdbReadService.ts` - Read operations
|
|
2. Create `services/cmdb/CmdbWriteService.ts` - Write operations with conflict detection
|
|
3. Create `services/cmdb/CmdbQueryService.ts` - Query operations
|
|
|
|
**Files to Create:**
|
|
- `services/cmdb/CmdbReadService.ts`
|
|
- `services/cmdb/CmdbWriteService.ts`
|
|
- `services/cmdb/CmdbQueryService.ts`
|
|
|
|
**Files to Modify:**
|
|
- `services/cmdbService.ts` - Split into read/write/query services
|
|
|
|
### Step 7: Extract Sync Service
|
|
|
|
**Goal:** Separate sync orchestration from storage
|
|
|
|
1. Create `domain/sync/SyncEngine.ts` - Sync business logic
|
|
2. Create `services/sync/SyncService.ts` - Sync orchestration
|
|
|
|
**Files to Create:**
|
|
- `domain/sync/SyncEngine.ts`
|
|
- `services/sync/SyncService.ts`
|
|
|
|
**Files to Modify:**
|
|
- `services/syncEngine.ts` - Split into domain + service
|
|
|
|
### Step 8: Refactor API Routes
|
|
|
|
**Goal:** Thin HTTP handlers delegating to services
|
|
|
|
1. Create `api/handlers/` directory
|
|
2. Move route logic to handlers
|
|
3. Routes become thin wrappers around handlers
|
|
|
|
**Files to Create:**
|
|
- `api/handlers/ObjectHandler.ts`
|
|
- `api/handlers/SchemaHandler.ts`
|
|
- `api/handlers/SyncHandler.ts`
|
|
|
|
**Files to Modify:**
|
|
- `routes/applications.ts` - Extract handlers
|
|
- `routes/objects.ts` - Extract handlers
|
|
- `routes/cache.ts` - Extract handlers
|
|
- `routes/schema.ts` - Extract handlers
|
|
|
|
### Step 9: Clean Up Legacy Code
|
|
|
|
**Goal:** Remove deprecated files
|
|
|
|
**Files to Delete:**
|
|
- `services/cacheStore.old.ts`
|
|
- Deprecated service files after migration complete
|
|
|
|
### Step 10: Update Type Generation
|
|
|
|
**Goal:** Simplify type generation workflow
|
|
|
|
1. Consolidate type generation logic
|
|
2. Add automatic type generation on schema discovery
|
|
3. Update documentation
|
|
|
|
**Files to Modify:**
|
|
- `scripts/generate-types-from-db.ts` - Enhance with auto-discovery
|
|
- `scripts/discover-schema.ts` - Auto-generate types after discovery
|
|
|
|
---
|
|
|
|
## Explicit Deletion List
|
|
|
|
### Phase 2 (After Migration Complete)
|
|
|
|
1. **`backend/src/services/cacheStore.old.ts`**
|
|
- Reason: Legacy implementation, replaced by `normalizedCacheStore.ts`
|
|
- Deprecation date: TBD
|
|
|
|
2. **`backend/src/services/jiraAssets.ts`** (after extracting business logic)
|
|
- Reason: API calls moved to infrastructure layer, business logic to services
|
|
- Replacement: `infrastructure/jira/JiraAssetsClient.ts` + `services/cmdb/Cmdb*Service.ts`
|
|
|
|
3. **`backend/src/services/schemaCacheService.ts`** (after consolidation)
|
|
- Reason: Merged into `services/schema/SchemaService.ts`
|
|
|
|
4. **`backend/src/services/schemaMappingService.ts`** (after consolidation)
|
|
- Reason: Merged into `services/schema/SchemaService.ts`
|
|
|
|
5. **`backend/scripts/generate-schema.ts`** (if still present)
|
|
- Reason: Replaced by `generate-types-from-db.ts`
|
|
|
|
### Notes
|
|
|
|
- Keep old files until migration is complete and tested
|
|
- Mark as deprecated with `@deprecated` JSDoc comments
|
|
- Add migration guide for each deprecated file
|
|
|
|
---
|
|
|
|
## API Payload Contract & Recursion Insights
|
|
|
|
### Jira Assets API Payload Structure
|
|
|
|
The Jira Assets API returns objects with the following nested structure:
|
|
|
|
```typescript
|
|
interface JiraAssetsSearchResponse {
|
|
objectEntries: JiraAssetsObject[]; // Top-level array of objects
|
|
// ... pagination metadata
|
|
}
|
|
|
|
interface JiraAssetsObject {
|
|
id: number;
|
|
objectKey: string;
|
|
label: string;
|
|
objectType: {
|
|
id: number;
|
|
name: string;
|
|
};
|
|
attributes: JiraAssetsAttribute[]; // Array of attributes
|
|
updated?: string;
|
|
created?: string;
|
|
}
|
|
|
|
interface JiraAssetsAttribute {
|
|
objectTypeAttributeId: number;
|
|
objectTypeAttribute?: {
|
|
id: number;
|
|
name: string;
|
|
};
|
|
objectAttributeValues: Array<{ // Union type of value representations
|
|
value?: string; // For scalar values (text, number, etc.)
|
|
displayValue?: string; // Human-readable value
|
|
referencedObject?: { // For reference attributes
|
|
id: number;
|
|
objectKey: string;
|
|
label: string;
|
|
// ⚠️ CRITICAL: referencedObject may include attributes (level 2)
|
|
attributes?: JiraAssetsAttribute[]; // Recursive structure
|
|
};
|
|
status?: { // For status attributes
|
|
name: string;
|
|
};
|
|
// ... other type-specific fields
|
|
}>;
|
|
}
|
|
```
|
|
|
|
### Key Insights
|
|
|
|
#### 1. Recursive Structure
|
|
|
|
**Issue:** `referencedObject` may include `attributes[]` array (level 2 recursion).
|
|
|
|
**Current Handling:**
|
|
- `jiraAssetsClient.ts` uses `includeAttributesDeep=2` parameter
|
|
- This causes referenced objects to include their attributes
|
|
- Referenced objects' attributes may themselves contain referenced objects (level 3, 4, etc.)
|
|
- **Cycles are possible** (Object A references Object B, Object B references Object A)
|
|
|
|
**Current Code Location:**
|
|
- `backend/src/services/jiraAssetsClient.ts:222` - `includeAttributesDeep=2`
|
|
- `backend/src/services/jiraAssetsClient.ts:259-260` - Search with deep attributes
|
|
- `backend/src/services/jiraAssetsClient.ts:285-286` - POST search with deep attributes
|
|
|
|
**Impact:**
|
|
- Response payloads can be very large (deeply nested)
|
|
- Memory usage increases with depth
|
|
- Parsing becomes more complex
|
|
|
|
#### 2. Shallow Referenced Objects
|
|
|
|
**Issue:** When `attributes[]` is absent on a shallow `referencedObject`, **do not wipe attributes**.
|
|
|
|
**Current Behavior:**
|
|
- Some code paths may clear attributes if `attributes` is missing
|
|
- This is incorrect - absence of `attributes` array does not mean "no attributes"
|
|
- It simply means "attributes not included in this response"
|
|
|
|
**Critical Rule:**
|
|
```typescript
|
|
// ❌ WRONG: Don't do this
|
|
if (!referencedObject.attributes) {
|
|
referencedObject.attributes = []; // This wipes existing attributes!
|
|
}
|
|
|
|
// ✅ CORRECT: Preserve existing attributes if missing from response
|
|
if (referencedObject.attributes === undefined) {
|
|
// Don't modify - attributes simply not included in this response
|
|
// Keep any existing attributes that were previously loaded
|
|
}
|
|
```
|
|
|
|
**Code Locations to Review:**
|
|
- `backend/src/services/jiraAssetsClient.ts:parseObject()` - Object parsing
|
|
- `backend/src/services/jiraAssetsClient.ts:parseAttributeValue()` - Reference parsing
|
|
- `backend/src/services/normalizedCacheStore.ts:loadAttributeValues()` - Reference reconstruction
|
|
|
|
#### 3. Attribute Values Union Type
|
|
|
|
**Issue:** `objectAttributeValues` is a union type - different value representations based on attribute type.
|
|
|
|
**Value Types:**
|
|
- Scalar (text, number, boolean): `{ value?: string, displayValue?: string }`
|
|
- Reference: `{ referencedObject?: { id, objectKey, label, attributes? } }`
|
|
- Status: `{ status?: { name: string } }`
|
|
- Date/Datetime: `{ value?: string }` (ISO string)
|
|
|
|
**Current Handling:**
|
|
- `jiraAssetsClient.ts:parseAttributeValue()` uses switch on `attrDef.type`
|
|
- Different parsing logic for each type
|
|
- Reference types extract `referencedObject`, others use `value` or `displayValue`
|
|
|
|
**Code Location:**
|
|
- `backend/src/services/jiraAssetsClient.ts:521-628` - `parseAttributeValue()` method
|
|
|
|
#### 4. Cycles and Recursion Depth
|
|
|
|
**Issue:** Recursive references can create cycles.
|
|
|
|
**Examples:**
|
|
- Application A references Team X
|
|
- Team X references Application A (via some attribute)
|
|
- This creates a cycle at depth 2
|
|
|
|
**Current Handling:**
|
|
- No explicit cycle detection
|
|
- `includeAttributesDeep=2` limits depth but doesn't prevent cycles
|
|
- Potential for infinite loops during reconstruction
|
|
|
|
**Recommendation:**
|
|
- Add cycle detection during object reconstruction
|
|
- Use visited set to track processed object IDs
|
|
- Limit recursion depth explicitly (not just via API parameter)
|
|
|
|
**Code Locations:**
|
|
- `backend/src/services/normalizedCacheStore.ts:loadAttributeValues()` - Reference resolution
|
|
- `backend/src/services/normalizedCacheStore.ts:reconstructObject()` - Object reconstruction
|
|
|
|
### Refactoring Considerations
|
|
|
|
1. **Create dedicated parser module** for handling recursive payloads
|
|
2. **Add cycle detection** utility
|
|
3. **Separate shallow vs deep parsing** logic
|
|
4. **Preserve attribute state** when attributes array is absent
|
|
5. **Document recursion depth limits** clearly
|
|
|
|
---
|
|
|
|
## Appendix: Module Dependency Graph
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ API Routes │
|
|
│ (applications.ts, objects.ts, cache.ts, schema.ts) │
|
|
└────────────┬────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Application Services │
|
|
│ (cmdbService.ts, dataService.ts) │
|
|
└────────────┬────────────────────────────────────────────┘
|
|
│
|
|
┌────────┴────────┐
|
|
▼ ▼
|
|
┌────────────┐ ┌──────────────────────────────┐
|
|
│ Sync │ │ Normalized Cache Store │
|
|
│ Engine │ │ (EAV Pattern) │
|
|
└─────┬──────┘ └──────────┬───────────────────┘
|
|
│ │
|
|
│ ▼
|
|
│ ┌─────────────────┐
|
|
│ │ Query Builder │
|
|
│ └─────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────┐
|
|
│ Jira Assets Client │
|
|
│ (jiraAssetsClient.ts, jiraAssets.ts) │
|
|
└────────────┬────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────┐
|
|
│ Schema Services │
|
|
│ (schemaDiscovery, schemaCache, etc.) │
|
|
└─────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps (Phase 2)
|
|
|
|
1. Review and approve this plan
|
|
2. Create detailed task breakdown for each migration step
|
|
3. Set up feature branch for refactoring
|
|
4. Implement changes incrementally with tests
|
|
5. Update documentation as we go
|
|
|
|
---
|
|
|
|
**End of Phase 1 - Analysis Document**
|