restructure tasks

tasks/shieldai-unified-restructure/20-alert-correlation-router.md

@@ -0,0 +1,102 @@
+# 20. Backend Router — Alert Correlation & Normalization Engine
+
+meta:
+  id: shieldai-unified-restructure-20
+  feature: shieldai-unified-restructure
+  priority: P1
+  depends_on: [shieldai-unified-restructure-15, shieldai-unified-restructure-16, shieldai-unified-restructure-17, shieldai-unified-restructure-18, shieldai-unified-restructure-19]
+  tags: [backend, trpc, correlation, alerts, api]
+
+objective:
+- Build the tRPC router and service layer for the cross-service alert correlation and normalization engine. Port logic from `packages/correlation/` into a unified `correlation` router that aggregates alerts from all services into a unified threat view.
+
+deliverables:
+- `web/src/server/api/routers/correlation.ts` — Correlation router:
+  - `correlation.getAlerts` — `protectedProcedure` returning normalized alerts for user
+  - `correlation.getAlertDetails` — `protectedProcedure` returning single alert with correlated data
+  - `correlation.getGroups` — `protectedProcedure` returning correlation groups
+  - `correlation.getGroupDetails` — `protectedProcedure` returning group with all member alerts
+  - `correlation.resolveAlert` — `protectedProcedure` marking alert as resolved or false positive
+  - `correlation.getStats` — `protectedProcedure` returning alert statistics
+- `web/src/server/services/correlation.service.ts` — Core business logic:
+  - `normalizeAlert(source, sourceAlertId, category, severity, userId, title, description, entities)` — create NormalizedAlert
+  - `correlateAlerts(userId)` — group related alerts by shared entities (email, phone, SSN)
+  - `getAlertTimeline(userId, filters?)` — chronological view of all alerts
+  - `resolveAlert(alertId, resolution)` — mark as resolved or false positive
+  - `getThreatScore(userId)` — calculate overall threat score based on alert severity and frequency
+- `web/src/server/services/correlation/engine.ts` — Correlation engine:
+  - `findRelatedAlerts(alert)` — find alerts sharing entities with given alert
+  - `createCorrelationGroup(alerts)` — group related alerts into CorrelationGroup
+  - `updateGroupSeverity(group)` — recalculate highest severity for group
+  - `deduplicateAlerts(alerts)` — remove duplicate alerts based on sourceAlertId
+- `web/src/server/services/correlation/normalizer.ts` — Alert normalization:
+  - `normalizeDarkWatchAlert(exposure)` — convert exposure to NormalizedAlert
+  - `normalizeSpamShieldAlert(detection)` — convert spam detection to NormalizedAlert
+  - `normalizeVoicePrintAlert(analysis)` — convert voice analysis to NormalizedAlert
+  - `normalizeHomeTitleAlert(change)` — convert property change to NormalizedAlert
+  - `normalizeRemoveBrokersAlert(listing)` — convert broker listing to NormalizedAlert
+
+steps:
+1. Create `web/src/server/api/routers/correlation.ts`.
+2. Define Zod schemas:
+   - `alertFilterSchema`: `source: z.enum([...]).optional()`, `severity: z.enum([...]).optional()`, `status: z.enum([...]).optional()`, `page`, `limit`
+   - `resolveSchema`: `alertId: z.string().uuid()`, `resolution: z.enum(['RESOLVED', 'FALSE_POSITIVE'])`
+3. Implement router procedures:
+   - Alert listing with filtering and pagination
+   - Group listing and details
+   - Alert resolution with audit logging
+   - Stats aggregation
+4. Create `web/src/server/services/correlation.service.ts`:
+   - Port from `packages/correlation/src/`
+   - Implement normalization pipeline
+   - Implement correlation grouping
+5. Create correlation engine:
+   - `findRelatedAlerts`: query alerts by shared entities (email, phone, SSN) within time window
+   - `createCorrelationGroup`: create group record, link alerts
+   - `updateGroupSeverity`: aggregate severity of all alerts in group
+   - `deduplicateAlerts`: ensure no duplicate sourceAlertId in normalized table
+6. Create normalizer module:
+   - One function per service domain that converts domain-specific alert to NormalizedAlert
+   - Map domain severity to NormalizedAlertSeverity enum
+   - Extract entities (emails, phones, SSNs) from payload for correlation
+7. Implement threat scoring:
+   - Formula: weighted sum of alert severities over 30-day window
+   - Decay older alerts
+   - Return score 0-100
+8. Integrate with other services:
+   - Call `correlationService.normalizeAlert()` from each service's alert pipeline
+   - DarkWatch (task 15), VoicePrint (task 16), SpamShield (task 17), HomeTitle (task 18), RemoveBrokers (task 19)
+9. Wire router into `web/src/server/api/root.ts`.
+10. Write unit tests for engine functions.
+
+steps:
+- Unit: `normalizeAlert` creates correct NormalizedAlert for each source type
+- Unit: `findRelatedAlerts` groups alerts sharing an email address
+- Unit: `createCorrelationGroup` creates group with correct highest severity
+- Unit: `deduplicateAlerts` prevents duplicate sourceAlertId
+- Unit: `getThreatScore` returns higher score for more severe/recent alerts
+- Integration: tRPC `getAlerts` returns normalized alerts for authenticated user
+
+acceptance_criteria:
+- [ ] Alerts from all 5 services are normalized into a unified schema
+- [ ] Related alerts are grouped by shared entities (email, phone, SSN)
+- [ ] Correlation groups update their severity when new alerts are added
+- [ ] Users can resolve alerts or mark them as false positives
+- [ ] Alert timeline provides chronological view across all services
+- [ ] Threat score accurately reflects user's current risk level
+- [ ] Deduplication prevents duplicate alerts from the same source
+
+validation:
+- Create normalized alerts from different services with shared email
+- Verify correlation engine groups them into a single group
+- Resolve an alert and verify status updated in DB
+- Calculate threat score for a user with mixed alert severities
+- Run `cd web && pnpm test` for correlation unit tests
+
+notes:
+- Reference legacy: `packages/correlation/src/`, `packages/api/src/routes/correlation.routes.ts`
+- The correlation engine is the "brain" that makes ShieldAI feel unified. Invest time in getting the entity extraction and grouping logic right.
+- Entity extraction should use regex patterns for emails, phones, and SSNs. Consider using a library like `compromise` for NLP extraction if payloads are unstructured.
+- Time window for correlation: alerts within 30 days sharing an entity should be grouped. Adjust based on testing.
+- The threat score algorithm should be transparent to users. Consider showing the breakdown (e.g., "+20 from DarkWatch exposure, +15 from SpamShield detection").
+- False positive tracking is important for ML model improvement. Log all false positive marks with context.