Kordant/tasks/kordant-unified-restructure/22-background-jobs.md

22. Background Jobs — Scheduler, Scan Workers, and Reminders

meta: id: kordant-unified-restructure-22 feature: kordant-unified-restructure priority: P1 depends_on: [kordant-unified-restructure-20] tags: [backend, jobs, scheduler, workers, background]

objective:

• Build the background job system for the unified monolith. Port logic from packages/jobs/ and all service schedulers into a unified job queue and worker system that handles recurring scans, report generation, and reminder notifications.

deliverables:

• web/src/server/jobs/queue.ts — Job queue abstraction:
  • Interface for enqueueing, dequeuing, and marking jobs complete/failed
  • Implementation using bullmq with Redis, or lightweight in-memory queue for dev
  • Job types: darkwatch.scan, voiceprint.batch, hometitle.scan, removebrokers.process, reports.generate, notifications.send
• web/src/server/jobs/worker.ts — Worker process:
  • Polls queue for jobs
  • Dispatches to correct handler based on job type
  • Handles retries with exponential backoff
  • Logs all job execution
• web/src/server/jobs/handlers/ — Job handlers:
  • darkwatch.scan.ts — scheduled dark web scans per subscription tier
  • voiceprint.batch.ts — batch voice analysis jobs
  • hometitle.scan.ts — scheduled property record scans
  • removebrokers.process.ts — process pending removal requests
  • reports.generate.ts — scheduled report generation
  • notifications.send.ts — queued notification delivery
• web/src/server/jobs/scheduler.ts — Cron scheduler:
  • Registers recurring jobs based on schedules in DB
  • Tier-based frequencies:
    • Basic: DarkWatch monthly, HomeTitle monthly
    • Plus: DarkWatch weekly, HomeTitle weekly, Reports monthly
    • Premium: DarkWatch daily, HomeTitle daily, Reports weekly, real-time monitoring
  • Uses node-cron or bullmq repeatables
• web/src/server/jobs/index.ts — Entry point:
  • Initializes queue, scheduler, and workers
  • Graceful shutdown handling

steps:

1. Install dependencies in web/:
   • bullmq (Redis-based queue)
   • ioredis (Redis client)
   • node-cron (cron expressions)
2. Create web/src/server/jobs/queue.ts:
   • Define JobType enum
   • Define JobPayload union type per job type
   • enqueue(jobType, payload, options?) — add job to queue
   • dequeue() — get next job (used by worker)
   • markComplete(jobId), markFailed(jobId, error) — update status
3. Create web/src/server/jobs/worker.ts:
   • startWorker() — begin polling loop
   • processJob(job) — switch on job.type, call handler
   • stopWorker() — graceful shutdown
   • Retry logic: max 3 attempts, exponential backoff (1min, 5min, 15min)
4. Create job handlers:
   • Each handler is an async function taking payload and returning result
   • darkwatch.scan: query all active watchlists, run scan engine (task 15), create alerts
   • voiceprint.batch: process pending AnalysisJob records, run ML inference
   • hometitle.scan: query all watched properties, run county scans (task 18), detect changes
   • removebrokers.process: query PENDING requests, attempt removal, update status
   • reports.generate: compile data, render HTML, generate PDF, save to storage
   • notifications.send: send queued email/push/SMS via notification service (task 14)
5. Create web/src/server/jobs/scheduler.ts:
   • registerSchedules() — read SchedulerConfig from DB, create cron jobs
   • scheduleForSubscription(subscription) — create tier-appropriate schedules
   • removeSchedulesForSubscription(subscriptionId) — clean up on cancellation
6. Create web/src/server/jobs/index.ts:
   • Initialize Redis connection
   • Start scheduler
   • Start worker(s)
   • Handle SIGINT/SIGTERM for graceful shutdown
7. Integrate with tRPC:
   • Add scheduler.runJobNow admin procedure for manual job triggering
   • Add scheduler.getJobStatus procedure for checking job progress
8. Write unit tests for handlers with mocked services.

steps:

• Unit: enqueue adds job to queue with correct payload
• Unit: processJob dispatches to correct handler
• Unit: darkwatch.scan handler queries watchlists and triggers scans
• Unit: Retry logic attempts job 3 times with correct delays
• Integration: Scheduled job runs and creates expected DB records

acceptance_criteria:

• Jobs can be enqueued with type-specific payloads
• Worker processes jobs and dispatches to correct handler
• Failed jobs are retried up to 3 times with exponential backoff
• Scheduled scans run at tier-appropriate frequencies
• Manual job triggering works via admin API
• Job status is queryable (pending, running, completed, failed)
• Graceful shutdown completes in-flight jobs before exiting
• All job handlers are tested with mocked dependencies

validation:

• Enqueue a test job and verify it appears in queue
• Start worker and verify job is processed
• Cause a handler to throw and verify retry behavior
• Verify scheduled jobs are created for a Premium subscription
• Run cd web && pnpm test for job unit tests

notes:

• Reference legacy: packages/jobs/src/, services/*/src/scheduler.service.ts
• Redis is required for BullMQ. For local development without Redis, implement a simple in-memory queue fallback.
• The worker can run in the same process as the web server (dev) or as a separate process (production). Design for both.
• Job handlers should be idempotent: running the same job twice should not create duplicates.
• Consider using bullmq's built-in cron support instead of node-cron for better reliability.
• Monitor queue depth and job failure rates. Add alerting if queue grows unexpectedly.
• For high-volume operations (e.g., scanning thousands of watchlist items), consider batching jobs or using worker concurrency.