Kordant/tasks/shieldai-unified-restructure/19-removebrokers-router.md

# 19. Backend Router — RemoveBrokers (Data Broker Removal)

meta:
  id: shieldai-unified-restructure-19
  feature: shieldai-unified-restructure
  priority: P1
  depends_on: [shieldai-unified-restructure-12, shieldai-unified-restructure-13, shieldai-unified-restructure-14]
  tags: [backend, trpc, removebrokers, privacy, api]

objective:
- Build the tRPC router for RemoveBrokers, the data broker removal service. Port all logic from `services/removebrokers/` and `packages/api/src/routes/removebrokers.routes.ts` into a unified `removebrokers` router and service layer.

deliverables:
- `web/src/server/api/routers/removebrokers.ts` — RemoveBrokers router:
  - `removebrokers.getBrokerRegistry` — `protectedProcedure` returning list of supported brokers
  - `removebrokers.getRemovalRequests` — `protectedProcedure` returning user's removal requests
  - `removebrokers.createRemovalRequest` — `protectedProcedure` initiating removal for a broker
  - `removebrokers.getRequestStatus` — `protectedProcedure` checking removal progress
  - `removebrokers.getBrokerListings` — `protectedProcedure` returning found listings
  - `removebrokers.scanForListings` — `protectedProcedure` scanning brokers for user's data
  - `removebrokers.getStats` — `protectedProcedure` returning removal statistics
- `web/src/server/services/removebrokers.service.ts` — Core business logic:
  - `getBrokerRegistry()` — return all active brokers with metadata
  - `createRemovalRequest(subscriptionId, brokerId, personalInfo)` — validate, create request, initiate removal
  - `getRequestStatus(requestId)` — return current status and history
  - `scanForListings(subscriptionId, brokerId)` — search broker site for user's data
  - `processRemovals()` — batch processor for pending removals
  - `updateRequestStatus(requestId, status, metadata)` — update after broker response
- `web/src/server/services/removebrokers/broker.registry.ts` — Broker definitions:
  - Static registry of supported data brokers
  - Each entry: name, domain, category, removalMethod, removalUrl, requiresAccount, estimatedDays
  - Methods: AUTOMATED, MANUAL_FORM, EMAIL, PHONE, MAIL
- `web/src/server/services/removebrokers/removal.engine.ts` — Removal automation:
  - `submitAutomatedRemoval(broker, personalInfo)` — API-based removal where available
  - `generateFormPayload(broker, personalInfo)` — prepare form data for manual submission
  - `sendRemovalEmail(broker, personalInfo)` — email-based removal request
  - `trackRemovalStatus(broker, requestId)` — poll broker for status updates

steps:
1. Create `web/src/server/api/routers/removebrokers.ts`.
2. Define Zod schemas:
   - `createRequestSchema`: `brokerId: z.string().uuid()`, `personalInfo: z.object({ fullName: z.string(), email: z.string().optional(), phone: z.string().optional(), address: z.string().optional(), dob: z.string().optional() })`
   - `scanSchema`: `brokerId: z.string().uuid().optional()` (if omitted, scan all)
3. Implement router procedures:
   - Broker registry listing (public or protected)
   - Removal request CRUD
   - Listing scan and results
   - Stats aggregation
4. Create `web/src/server/services/removebrokers.service.ts`:
   - Port from `services/removebrokers/src/`
   - Implement request lifecycle: PENDING → SUBMITTED → IN_PROGRESS → COMPLETED/FAILED
5. Create broker registry:
   - Hardcode initial list of 20-50 major data brokers
   - Include removal instructions and URLs
   - Allow admin updates via API (future enhancement)
6. Create removal engine:
   - `submitAutomatedRemoval`: call broker API if available (rare)
   - `generateFormPayload`: create structured data for form filling
   - `sendRemovalEmail`: use notification service (task 14) to send removal request email
   - `trackRemovalStatus`: placeholder for polling logic
7. Implement listing scanner:
   - Search broker websites for user's name, email, phone
   - Use web scraping (Cheerio, Playwright) where APIs are unavailable
   - Store found listings in `BrokerListing` table
8. Implement scheduler integration:
   - Pending removals should be picked up by background job (task 22)
   - Retries for failed removals with exponential backoff
9. Wire router into `web/src/server/api/root.ts`.
10. Write unit tests with mocked broker interactions.

steps:
- Unit: `createRemovalRequest` creates record with PENDING status
- Unit: `processRemovals` advances eligible requests to SUBMITTED
- Unit: `scanForListings` creates BrokerListing records for found data
- Unit: Broker registry returns correct metadata for known brokers
- Integration: tRPC procedures enforce subscription scoping

acceptance_criteria:
- [ ] Broker registry lists all supported data brokers with removal methods
- [ ] Removal requests can be created per broker with personal info
- [ ] Request status tracks lifecycle from PENDING to COMPLETED/FAILED
- [ ] Listings scanner finds user's data on broker sites
- [ ] Automated removals use APIs where available; manual methods generate instructions
- [ ] Failed removals are retried with exponential backoff
- [ ] Stats endpoint shows removal progress (total, completed, pending, failed)

validation:
- List brokers and verify registry data
- Create a removal request and verify DB record
- Simulate a scan and verify listing records created
- Run `cd web && pnpm test` for RemoveBrokers unit tests

notes:
- Reference legacy: `services/removebrokers/src/`, `packages/api/src/routes/removebrokers.routes.ts`
- Data broker removal is often a manual process. The automation layer should handle the easy cases and provide clear instructions for manual cases.
- Web scraping broker sites may violate Terms of Service. Use public APIs where available and document legal considerations.
- Personal info submitted for removal should be handled carefully. Do not log raw PII.
- The removal engine should be designed as a plugin system: each broker can have its own adapter for API, form, or email removal.
- Consider integrating with a third-party service (e.g., DeleteMe, Optery) for broader broker coverage if building individual adapters is impractical.