Consolidate documentation and update backend services

- Reorganize docs into 'Core deployment guides' and 'Setup and configuration' subdirectories - Consolidate redundant documentation files (ACR, pipelines, deployment guides) - Add documentation consolidation plan - Update backend database factory and logger services - Update migration script and docker-compose configurations - Add PostgreSQL setup script
2026-01-22 22:45:54 +01:00
parent 18aec4ad80
commit f4399a8e4e
49 changed files with 1320 additions and 7243 deletions
--- a/guides/GREEN-FIELD-DEPLOYMENT-GUIDE.md
+++ b/guides/GREEN-FIELD-DEPLOYMENT-GUIDE.md
@@ -0,0 +1,592 @@
+# Green Field Deployment Guide
+
+## Overzicht
+
+Deze guide beschrijft hoe je de applicatie volledig opnieuw deployt met de nieuwe genormaliseerde database structuur. Aangezien het een green field deployment is, kunnen we alles schoon opzetten.
+
+---
+
+## Stap 1: Database Setup
+
+### Option A: PostgreSQL (Aanbevolen voor productie)
+
+#### 1.1 Azure Database for PostgreSQL
+
+```bash
+# Via Azure Portal of CLI
+az postgres flexible-server create \
+  --resource-group <resource-group> \
+  --name <server-name> \
+  --location <location> \
+  --admin-user <admin-user> \
+  --admin-password <admin-password> \
+  --sku-name Standard_B1ms \
+  --tier Burstable \
+  --version 14
+```
+
+#### 1.2 Database Aanmaken
+
+**Note:** The application uses a single database for all data. All tables (CMDB cache, classification history, and session state) are stored in the same database.
+
+```sql
+-- Connect to PostgreSQL
+CREATE DATABASE cmdb_insight;
+
+-- Create user (optional, can use admin user)
+CREATE USER cmdb_user WITH PASSWORD 'secure_password';
+GRANT ALL PRIVILEGES ON DATABASE cmdb_insight TO cmdb_user;
+```
+
+#### 1.3 Connection String
+
+```env
+DATABASE_TYPE=postgres
+DATABASE_URL=postgresql://cmdb_user:secure_password@<server-name>.postgres.database.azure.com:5432/cmdb_insight?sslmode=require
+```
+
+### Option B: SQLite (Voor development/testing)
+
+```env
+DATABASE_TYPE=sqlite
+# Database files worden automatisch aangemaakt in backend/data/
+```
+
+---
+
+## Stap 2: Environment Variables
+
+### 2.1 Basis Configuratie
+
+Maak `.env` bestand in project root:
+
+```env
+# Server
+PORT=3001
+NODE_ENV=production
+FRONTEND_URL=https://your-domain.com
+
+# Database (zie Stap 1)
+DATABASE_TYPE=postgres
+DATABASE_URL=postgresql://...
+
+# Jira Assets
+JIRA_HOST=https://jira.zuyderland.nl
+JIRA_SCHEMA_ID=<your_schema_id>
+JIRA_SERVICE_ACCOUNT_TOKEN=<service_account_token>
+
+# Jira Authentication Method
+JIRA_AUTH_METHOD=oauth
+
+# OAuth Configuration (als JIRA_AUTH_METHOD=oauth)
+JIRA_OAUTH_CLIENT_ID=<client_id>
+JIRA_OAUTH_CLIENT_SECRET=<client_secret>
+JIRA_OAUTH_CALLBACK_URL=https://your-domain.com/api/auth/callback
+JIRA_OAUTH_SCOPES=READ WRITE
+
+# Session
+SESSION_SECRET=<generate_secure_random_string>
+
+# AI (configureer per gebruiker in profile settings)
+# ANTHROPIC_API_KEY, OPENAI_API_KEY, TAVILY_API_KEY worden per gebruiker ingesteld
+```
+
+### 2.2 Session Secret Genereren
+
+```bash
+# Generate secure random string
+node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
+```
+
+---
+
+## Stap 3: Schema Discovery
+
+### 3.1 Schema Genereren
+
+```bash
+cd backend
+npm run generate-schema
+```
+
+Dit:
+- Haalt schema op van Jira Assets API
+- Genereert `backend/src/generated/jira-schema.ts`
+- Genereert `backend/src/generated/jira-types.ts`
+
+### 3.2 Schema Populeren in Database
+
+Bij eerste start van de applicatie:
+- `schemaDiscoveryService` leest `OBJECT_TYPES` uit generated schema
+- Populeert `object_types` en `attributes` tabellen
+- Gebeurt automatisch bij initialisatie
+
+---
+
+## Stap 4: Application Build
+
+### 4.1 Dependencies Installeren
+
+```bash
+# Root
+npm install
+
+# Backend
+cd backend
+npm install
+
+# Frontend
+cd frontend
+npm install
+```
+
+### 4.2 Build
+
+```bash
+# Backend
+cd backend
+npm run build
+
+# Frontend
+cd frontend
+npm run build
+```
+
+---
+
+## Stap 5: Database Initialisatie
+
+### 5.1 Automatische Initialisatie
+
+Bij eerste start:
+1. Normalized schema wordt aangemaakt
+2. Schema discovery wordt uitgevoerd
+3. Tabellen worden gevuld met object types en attributes
+
+### 5.2 Handmatige Verificatie (Optioneel)
+
+```sql
+-- Check object types
+SELECT * FROM object_types ORDER BY sync_priority;
+
+-- Check attributes
+SELECT COUNT(*) FROM attributes;
+
+-- Check per type
+SELECT object_type_name, COUNT(*) as attr_count 
+FROM attributes 
+GROUP BY object_type_name;
+```
+
+---
+
+## Stap 6: Data Sync
+
+### 6.1 Eerste Sync
+
+```bash
+# Via API (na deployment)
+curl -X POST https://your-domain.com/api/cache/sync \
+  -H "Authorization: Bearer <token>"
+```
+
+Of via applicatie:
+- Ga naar Settings → Cache Management
+- Klik "Full Sync"
+
+### 6.2 Sync Status Checken
+
+```bash
+curl https://your-domain.com/api/cache/status \
+  -H "Authorization: Bearer <token>"
+```
+
+---
+
+## Stap 7: Docker Deployment
+
+### 7.1 Build Images
+
+```bash
+# Backend
+docker build -t cmdb-backend:latest -f backend/Dockerfile .
+
+# Frontend
+docker build -t cmdb-frontend:latest -f frontend/Dockerfile .
+```
+
+### 7.2 Docker Compose (Production)
+
+```yaml
+# docker-compose.prod.yml
+version: '3.8'
+
+services:
+  backend:
+    image: cmdb-backend:latest
+    environment:
+      - DATABASE_TYPE=postgres
+      - DATABASE_URL=${DATABASE_URL}
+      - JIRA_HOST=${JIRA_HOST}
+      - JIRA_SCHEMA_ID=${JIRA_SCHEMA_ID}
+      - JIRA_SERVICE_ACCOUNT_TOKEN=${JIRA_SERVICE_ACCOUNT_TOKEN}
+      - SESSION_SECRET=${SESSION_SECRET}
+    ports:
+      - "3001:3001"
+
+  frontend:
+    image: cmdb-frontend:latest
+    ports:
+      - "80:80"
+    depends_on:
+      - backend
+
+  nginx:
+    image: nginx:alpine
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
+    ports:
+      - "443:443"
+    depends_on:
+      - frontend
+      - backend
+```
+
+### 7.3 Starten
+
+```bash
+docker-compose -f docker-compose.prod.yml up -d
+```
+
+---
+
+## Stap 8: Azure App Service Deployment
+
+### 8.1 Azure Container Registry
+
+```bash
+# Login
+az acr login --name <registry-name>
+
+# Tag images
+docker tag cmdb-backend:latest <registry-name>.azurecr.io/cmdb-backend:latest
+docker tag cmdb-frontend:latest <registry-name>.azurecr.io/cmdb-frontend:latest
+
+# Push
+docker push <registry-name>.azurecr.io/cmdb-backend:latest
+docker push <registry-name>.azurecr.io/cmdb-frontend:latest
+```
+
+### 8.2 App Service Configuratie
+
+**Backend App Service:**
+- Container: `<registry-name>.azurecr.io/cmdb-backend:latest`
+- Environment variables: Alle `.env` variabelen
+- Port: 3001
+
+**Frontend App Service:**
+- Container: `<registry-name>.azurecr.io/cmdb-frontend:latest`
+- Environment variables: `VITE_API_URL=https://backend-app.azurewebsites.net`
+
+### 8.3 Deployment via Azure DevOps
+
+Zie `azure-pipelines.yml` voor CI/CD pipeline.
+
+---
+
+## Stap 9: Verificatie
+
+### 9.1 Health Checks
+
+```bash
+# Backend health
+curl https://backend-app.azurewebsites.net/health
+
+# Frontend
+curl https://frontend-app.azurewebsites.net
+```
+
+### 9.2 Database Verificatie
+
+```sql
+-- Check object count
+SELECT object_type_name, COUNT(*) as count 
+FROM objects 
+GROUP BY object_type_name;
+
+-- Check attribute values
+SELECT COUNT(*) FROM attribute_values;
+
+-- Check relations
+SELECT COUNT(*) FROM object_relations;
+```
+
+### 9.3 Functionaliteit Testen
+
+1. **Login** - Test authenticatie
+2. **Dashboard** - Check of data wordt getoond
+3. **Application List** - Test filters
+4. **Application Detail** - Test edit functionaliteit
+5. **Sync** - Test manual sync
+
+---
+
+## Stap 10: Monitoring & Maintenance
+
+### 10.1 Logs
+
+```bash
+# Azure App Service logs
+az webapp log tail --name <app-name> --resource-group <resource-group>
+
+# Docker logs
+docker-compose logs -f backend
+```
+
+### 10.2 Database Monitoring
+
+```sql
+-- Database size (single database contains all data)
+SELECT pg_database_size('cmdb_insight');
+
+-- Table sizes
+SELECT 
+  schemaname,
+  tablename,
+  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
+FROM pg_tables
+WHERE schemaname = 'public'
+ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
+
+-- Index usage
+SELECT 
+  schemaname,
+  tablename,
+  indexname,
+  idx_scan as index_scans
+FROM pg_stat_user_indexes
+ORDER BY idx_scan DESC;
+```
+
+### 10.3 Performance Monitoring
+
+- Monitor query performance
+- Check sync duration
+- Monitor database connections
+- Check memory usage
+
+---
+
+## Troubleshooting
+
+### Database Connection Issues
+
+```bash
+# Test connection
+psql $DATABASE_URL -c "SELECT 1"
+
+# Check firewall rules (Azure)
+az postgres flexible-server firewall-rule list \
+  --resource-group <resource-group> \
+  --name <server-name>
+```
+
+### Schema Discovery Fails
+
+```bash
+# Check Jira connection
+curl -H "Authorization: Bearer $JIRA_SERVICE_ACCOUNT_TOKEN" \
+  $JIRA_HOST/rest/insight/1.0/objectschema/list
+
+# Regenerate schema
+cd backend
+npm run generate-schema
+```
+
+### Sync Issues
+
+```bash
+# Check sync status
+curl https://your-domain.com/api/cache/status
+
+# Manual sync for specific type
+curl -X POST https://your-domain.com/api/cache/sync/ApplicationComponent
+```
+
+---
+
+## Rollback Plan
+
+Als er problemen zijn:
+
+1. **Stop applicatie**
+2. **Revert code** (git)
+3. **Herstart applicatie**
+
+Aangezien het green field is, is er geen data migration nodig voor rollback.
+
+---
+
+## Post-Deployment Checklist
+
+- [ ] Database verbinding werkt
+- [ ] Schema discovery succesvol
+- [ ] Eerste sync voltooid
+- [ ] Alle object types gesynct
+- [ ] Queries werken correct
+- [ ] Filters werken
+- [ ] Edit functionaliteit werkt
+- [ ] Authentication werkt
+- [ ] Logs zijn zichtbaar
+- [ ] Monitoring is ingesteld
+
+---
+
+## Performance Tips
+
+1. **Database Indexes** - Zijn automatisch aangemaakt
+2. **Connection Pooling** - PostgreSQL adapter gebruikt pool (max 20)
+3. **Query Optimization** - Gebruik `queryWithFilters()` voor gefilterde queries
+4. **Sync Frequency** - Incremental sync elke 30 seconden (configureerbaar)
+
+---
+
+## Security Checklist
+
+- [ ] `SESSION_SECRET` is sterk en uniek
+- [ ] Database credentials zijn secure
+- [ ] HTTPS is ingeschakeld
+- [ ] CORS is correct geconfigureerd
+- [ ] OAuth callback URL is correct
+- [ ] Environment variables zijn niet gecommit
+
+---
+
+## Extra Tips & Best Practices
+
+### Database Performance
+
+1. **Connection Pooling**
+   - PostgreSQL adapter gebruikt automatisch connection pooling (max 20)
+   - Monitor pool usage in production
+
+2. **Query Optimization**
+   - Gebruik `queryWithFilters()` voor gefilterde queries (veel sneller)
+   - Indexes zijn automatisch aangemaakt
+   - Monitor slow queries
+
+3. **Sync Performance**
+   - Batch size: 50 objects per batch (configureerbaar via `JIRA_API_BATCH_SIZE`)
+   - Incremental sync: elke 30 seconden (configureerbaar via `SYNC_INCREMENTAL_INTERVAL_MS`)
+
+### Monitoring
+
+1. **Application Logs**
+   - Check voor schema discovery errors
+   - Monitor sync duration
+   - Check query performance
+
+2. **Database Metrics**
+   - Table sizes groeien
+   - Index usage
+   - Connection pool usage
+
+3. **Jira API**
+   - Monitor rate limiting
+   - Check API response times
+   - Monitor sync success rate
+
+### Backup Strategy
+
+1. **Database Backups**
+   - Azure PostgreSQL: automatische dagelijkse backups
+   - SQLite: maak periodieke copies van `.db` files
+
+2. **Configuration Backup**
+   - Backup `.env` file (securely!)
+   - Document alle environment variables
+
+### Scaling Considerations
+
+1. **Database**
+   - PostgreSQL kan schalen (vertical scaling)
+   - Overweeg read replicas voor grote datasets
+
+2. **Application**
+   - Stateless design - kan horizontaal schalen
+   - Session storage in database (scalable)
+
+3. **Cache**
+   - Normalized structure is efficient
+   - Indexes zorgen voor goede performance
+
+### Troubleshooting Common Issues
+
+#### Issue: Schema Discovery Fails
+
+**Symptoom:** Error bij startup, geen object types in database
+
+**Oplossing:**
+```bash
+# Check Jira connection
+curl -H "Authorization: Bearer $JIRA_SERVICE_ACCOUNT_TOKEN" \
+  $JIRA_HOST/rest/insight/1.0/objectschema/list
+
+# Regenerate schema
+cd backend
+npm run generate-schema
+
+# Restart application
+```
+
+#### Issue: Sync is Slow
+
+**Symptoom:** Full sync duurt lang
+
+**Oplossing:**
+- Check Jira API response times
+- Verhoog batch size (maar niet te veel - rate limiting)
+- Check database connection pool
+
+#### Issue: Queries zijn Langzaam
+
+**Symptoom:** Filters werken traag
+
+**Oplossing:**
+- Check of indexes bestaan: `\d+ attribute_values` in PostgreSQL
+- Gebruik `queryWithFilters()` in plaats van JavaScript filtering
+- Check query execution plan
+
+#### Issue: Memory Usage Hoog
+
+**Symptoom:** Applicatie gebruikt veel geheugen
+
+**Oplossing:**
+- Normalized storage gebruikt minder geheugen dan JSONB
+- Check of oude cacheStore nog ergens gebruikt wordt
+- Monitor object reconstruction (kan N+1 queries veroorzaken)
+
+### Development vs Production
+
+**Development:**
+- SQLite is prima voor testing
+- Lokale database in `backend/data/`
+- Geen SSL nodig
+
+**Production:**
+- PostgreSQL is aanbevolen
+- Azure Database for PostgreSQL
+- SSL vereist
+- Connection pooling
+- Monitoring ingeschakeld
+
+### Migration van Development naar Production
+
+1. **Schema is hetzelfde** - geen migratie nodig
+2. **Data sync** - eerste sync haalt alles op van Jira
+3. **Environment variables** - update voor productie
+4. **OAuth callback URL** - update naar productie domain
+
+---
+
+**End of Guide**