restructure tasks

tasks/shieldai-unified-restructure/22-background-jobs.md

@@ -0,0 +1,110 @@
+# 22. Background Jobs — Scheduler, Scan Workers, and Reminders
+
+meta:
+  id: shieldai-unified-restructure-22
+  feature: shieldai-unified-restructure
+  priority: P1
+  depends_on: [shieldai-unified-restructure-20]
+  tags: [backend, jobs, scheduler, workers, background]
+
+objective:
+- Build the background job system for the unified monolith. Port logic from `packages/jobs/` and all service schedulers into a unified job queue and worker system that handles recurring scans, report generation, and reminder notifications.
+
+deliverables:
+- `web/src/server/jobs/queue.ts` — Job queue abstraction:
+  - Interface for enqueueing, dequeuing, and marking jobs complete/failed
+  - Implementation using `bullmq` with Redis, or lightweight in-memory queue for dev
+  - Job types: `darkwatch.scan`, `voiceprint.batch`, `hometitle.scan`, `removebrokers.process`, `reports.generate`, `notifications.send`
+- `web/src/server/jobs/worker.ts` — Worker process:
+  - Polls queue for jobs
+  - Dispatches to correct handler based on job type
+  - Handles retries with exponential backoff
+  - Logs all job execution
+- `web/src/server/jobs/handlers/` — Job handlers:
+  - `darkwatch.scan.ts` — scheduled dark web scans per subscription tier
+  - `voiceprint.batch.ts` — batch voice analysis jobs
+  - `hometitle.scan.ts` — scheduled property record scans
+  - `removebrokers.process.ts` — process pending removal requests
+  - `reports.generate.ts` — scheduled report generation
+  - `notifications.send.ts` — queued notification delivery
+- `web/src/server/jobs/scheduler.ts` — Cron scheduler:
+  - Registers recurring jobs based on schedules in DB
+  - Tier-based frequencies:
+    - Basic: DarkWatch monthly, HomeTitle monthly
+    - Plus: DarkWatch weekly, HomeTitle weekly, Reports monthly
+    - Premium: DarkWatch daily, HomeTitle daily, Reports weekly, real-time monitoring
+  - Uses `node-cron` or `bullmq` repeatables
+- `web/src/server/jobs/index.ts` — Entry point:
+  - Initializes queue, scheduler, and workers
+  - Graceful shutdown handling
+
+steps:
+1. Install dependencies in `web/`:
+   - `bullmq` (Redis-based queue)
+   - `ioredis` (Redis client)
+   - `node-cron` (cron expressions)
+2. Create `web/src/server/jobs/queue.ts`:
+   - Define `JobType` enum
+   - Define `JobPayload` union type per job type
+   - `enqueue(jobType, payload, options?)` — add job to queue
+   - `dequeue()` — get next job (used by worker)
+   - `markComplete(jobId)`, `markFailed(jobId, error)` — update status
+3. Create `web/src/server/jobs/worker.ts`:
+   - `startWorker()` — begin polling loop
+   - `processJob(job)` — switch on `job.type`, call handler
+   - `stopWorker()` — graceful shutdown
+   - Retry logic: max 3 attempts, exponential backoff (1min, 5min, 15min)
+4. Create job handlers:
+   - Each handler is an async function taking `payload` and returning result
+   - `darkwatch.scan`: query all active watchlists, run scan engine (task 15), create alerts
+   - `voiceprint.batch`: process pending `AnalysisJob` records, run ML inference
+   - `hometitle.scan`: query all watched properties, run county scans (task 18), detect changes
+   - `removebrokers.process`: query PENDING requests, attempt removal, update status
+   - `reports.generate`: compile data, render HTML, generate PDF, save to storage
+   - `notifications.send`: send queued email/push/SMS via notification service (task 14)
+5. Create `web/src/server/jobs/scheduler.ts`:
+   - `registerSchedules()` — read `SchedulerConfig` from DB, create cron jobs
+   - `scheduleForSubscription(subscription)` — create tier-appropriate schedules
+   - `removeSchedulesForSubscription(subscriptionId)` — clean up on cancellation
+6. Create `web/src/server/jobs/index.ts`:
+   - Initialize Redis connection
+   - Start scheduler
+   - Start worker(s)
+   - Handle SIGINT/SIGTERM for graceful shutdown
+7. Integrate with tRPC:
+   - Add `scheduler.runJobNow` admin procedure for manual job triggering
+   - Add `scheduler.getJobStatus` procedure for checking job progress
+8. Write unit tests for handlers with mocked services.
+
+steps:
+- Unit: `enqueue` adds job to queue with correct payload
+- Unit: `processJob` dispatches to correct handler
+- Unit: `darkwatch.scan` handler queries watchlists and triggers scans
+- Unit: Retry logic attempts job 3 times with correct delays
+- Integration: Scheduled job runs and creates expected DB records
+
+acceptance_criteria:
+- [ ] Jobs can be enqueued with type-specific payloads
+- [ ] Worker processes jobs and dispatches to correct handler
+- [ ] Failed jobs are retried up to 3 times with exponential backoff
+- [ ] Scheduled scans run at tier-appropriate frequencies
+- [ ] Manual job triggering works via admin API
+- [ ] Job status is queryable (pending, running, completed, failed)
+- [ ] Graceful shutdown completes in-flight jobs before exiting
+- [ ] All job handlers are tested with mocked dependencies
+
+validation:
+- Enqueue a test job and verify it appears in queue
+- Start worker and verify job is processed
+- Cause a handler to throw and verify retry behavior
+- Verify scheduled jobs are created for a Premium subscription
+- Run `cd web && pnpm test` for job unit tests
+
+notes:
+- Reference legacy: `packages/jobs/src/`, `services/*/src/scheduler.service.ts`
+- Redis is required for BullMQ. For local development without Redis, implement a simple in-memory queue fallback.
+- The worker can run in the same process as the web server (dev) or as a separate process (production). Design for both.
+- Job handlers should be idempotent: running the same job twice should not create duplicates.
+- Consider using `bullmq`'s built-in cron support instead of `node-cron` for better reliability.
+- Monitor queue depth and job failure rates. Add alerting if queue grows unexpectedly.
+- For high-volume operations (e.g., scanning thousands of watchlist items), consider batching jobs or using worker concurrency.