Kordant/tasks/shieldai-unified-restructure/20-alert-correlation-router.md

20. Backend Router — Alert Correlation & Normalization Engine

meta: id: shieldai-unified-restructure-20 feature: shieldai-unified-restructure priority: P1 depends_on: [shieldai-unified-restructure-15, shieldai-unified-restructure-16, shieldai-unified-restructure-17, shieldai-unified-restructure-18, shieldai-unified-restructure-19] tags: [backend, trpc, correlation, alerts, api]

objective:

• Build the tRPC router and service layer for the cross-service alert correlation and normalization engine. Port logic from packages/correlation/ into a unified correlation router that aggregates alerts from all services into a unified threat view.

deliverables:

• web/src/server/api/routers/correlation.ts — Correlation router:
  • correlation.getAlerts — protectedProcedure returning normalized alerts for user
  • correlation.getAlertDetails — protectedProcedure returning single alert with correlated data
  • correlation.getGroups — protectedProcedure returning correlation groups
  • correlation.getGroupDetails — protectedProcedure returning group with all member alerts
  • correlation.resolveAlert — protectedProcedure marking alert as resolved or false positive
  • correlation.getStats — protectedProcedure returning alert statistics
• web/src/server/services/correlation.service.ts — Core business logic:
  • normalizeAlert(source, sourceAlertId, category, severity, userId, title, description, entities) — create NormalizedAlert
  • correlateAlerts(userId) — group related alerts by shared entities (email, phone, SSN)
  • getAlertTimeline(userId, filters?) — chronological view of all alerts
  • resolveAlert(alertId, resolution) — mark as resolved or false positive
  • getThreatScore(userId) — calculate overall threat score based on alert severity and frequency
• web/src/server/services/correlation/engine.ts — Correlation engine:
  • findRelatedAlerts(alert) — find alerts sharing entities with given alert
  • createCorrelationGroup(alerts) — group related alerts into CorrelationGroup
  • updateGroupSeverity(group) — recalculate highest severity for group
  • deduplicateAlerts(alerts) — remove duplicate alerts based on sourceAlertId
• web/src/server/services/correlation/normalizer.ts — Alert normalization:
  • normalizeDarkWatchAlert(exposure) — convert exposure to NormalizedAlert
  • normalizeSpamShieldAlert(detection) — convert spam detection to NormalizedAlert
  • normalizeVoicePrintAlert(analysis) — convert voice analysis to NormalizedAlert
  • normalizeHomeTitleAlert(change) — convert property change to NormalizedAlert
  • normalizeRemoveBrokersAlert(listing) — convert broker listing to NormalizedAlert

steps:

1. Create web/src/server/api/routers/correlation.ts.
2. Define Zod schemas:
   • alertFilterSchema: source: z.enum([...]).optional(), severity: z.enum([...]).optional(), status: z.enum([...]).optional(), page, limit
   • resolveSchema: alertId: z.string().uuid(), resolution: z.enum(['RESOLVED', 'FALSE_POSITIVE'])
3. Implement router procedures:
   • Alert listing with filtering and pagination
   • Group listing and details
   • Alert resolution with audit logging
   • Stats aggregation
4. Create web/src/server/services/correlation.service.ts:
   • Port from packages/correlation/src/
   • Implement normalization pipeline
   • Implement correlation grouping
5. Create correlation engine:
   • findRelatedAlerts: query alerts by shared entities (email, phone, SSN) within time window
   • createCorrelationGroup: create group record, link alerts
   • updateGroupSeverity: aggregate severity of all alerts in group
   • deduplicateAlerts: ensure no duplicate sourceAlertId in normalized table
6. Create normalizer module:
   • One function per service domain that converts domain-specific alert to NormalizedAlert
   • Map domain severity to NormalizedAlertSeverity enum
   • Extract entities (emails, phones, SSNs) from payload for correlation
7. Implement threat scoring:
   • Formula: weighted sum of alert severities over 30-day window
   • Decay older alerts
   • Return score 0-100
8. Integrate with other services:
   • Call correlationService.normalizeAlert() from each service's alert pipeline
   • DarkWatch (task 15), VoicePrint (task 16), SpamShield (task 17), HomeTitle (task 18), RemoveBrokers (task 19)
9. Wire router into web/src/server/api/root.ts.
10. Write unit tests for engine functions.

steps:

• Unit: normalizeAlert creates correct NormalizedAlert for each source type
• Unit: findRelatedAlerts groups alerts sharing an email address
• Unit: createCorrelationGroup creates group with correct highest severity
• Unit: deduplicateAlerts prevents duplicate sourceAlertId
• Unit: getThreatScore returns higher score for more severe/recent alerts
• Integration: tRPC getAlerts returns normalized alerts for authenticated user

acceptance_criteria:

• Alerts from all 5 services are normalized into a unified schema
• Related alerts are grouped by shared entities (email, phone, SSN)
• Correlation groups update their severity when new alerts are added
• Users can resolve alerts or mark them as false positives
• Alert timeline provides chronological view across all services
• Threat score accurately reflects user's current risk level
• Deduplication prevents duplicate alerts from the same source

validation:

• Create normalized alerts from different services with shared email
• Verify correlation engine groups them into a single group
• Resolve an alert and verify status updated in DB
• Calculate threat score for a user with mixed alert severities
• Run cd web && pnpm test for correlation unit tests

notes:

• Reference legacy: packages/correlation/src/, packages/api/src/routes/correlation.routes.ts
• The correlation engine is the "brain" that makes ShieldAI feel unified. Invest time in getting the entity extraction and grouping logic right.
• Entity extraction should use regex patterns for emails, phones, and SSNs. Consider using a library like compromise for NLP extraction if payloads are unstructured.
• Time window for correlation: alerts within 30 days sharing an entity should be grouped. Adjust based on testing.
• The threat score algorithm should be transparent to users. Consider showing the breakdown (e.g., "+20 from DarkWatch exposure, +15 from SpamShield detection").
• False positive tracking is important for ML model improvement. Log all false positive marks with context.