restructure tasks

tasks/shieldai-unified-restructure/19-removebrokers-router.md

@@ -0,0 +1,99 @@
+# 19. Backend Router — RemoveBrokers (Data Broker Removal)
+
+meta:
+  id: shieldai-unified-restructure-19
+  feature: shieldai-unified-restructure
+  priority: P1
+  depends_on: [shieldai-unified-restructure-12, shieldai-unified-restructure-13, shieldai-unified-restructure-14]
+  tags: [backend, trpc, removebrokers, privacy, api]
+
+objective:
+- Build the tRPC router for RemoveBrokers, the data broker removal service. Port all logic from `services/removebrokers/` and `packages/api/src/routes/removebrokers.routes.ts` into a unified `removebrokers` router and service layer.
+
+deliverables:
+- `web/src/server/api/routers/removebrokers.ts` — RemoveBrokers router:
+  - `removebrokers.getBrokerRegistry` — `protectedProcedure` returning list of supported brokers
+  - `removebrokers.getRemovalRequests` — `protectedProcedure` returning user's removal requests
+  - `removebrokers.createRemovalRequest` — `protectedProcedure` initiating removal for a broker
+  - `removebrokers.getRequestStatus` — `protectedProcedure` checking removal progress
+  - `removebrokers.getBrokerListings` — `protectedProcedure` returning found listings
+  - `removebrokers.scanForListings` — `protectedProcedure` scanning brokers for user's data
+  - `removebrokers.getStats` — `protectedProcedure` returning removal statistics
+- `web/src/server/services/removebrokers.service.ts` — Core business logic:
+  - `getBrokerRegistry()` — return all active brokers with metadata
+  - `createRemovalRequest(subscriptionId, brokerId, personalInfo)` — validate, create request, initiate removal
+  - `getRequestStatus(requestId)` — return current status and history
+  - `scanForListings(subscriptionId, brokerId)` — search broker site for user's data
+  - `processRemovals()` — batch processor for pending removals
+  - `updateRequestStatus(requestId, status, metadata)` — update after broker response
+- `web/src/server/services/removebrokers/broker.registry.ts` — Broker definitions:
+  - Static registry of supported data brokers
+  - Each entry: name, domain, category, removalMethod, removalUrl, requiresAccount, estimatedDays
+  - Methods: AUTOMATED, MANUAL_FORM, EMAIL, PHONE, MAIL
+- `web/src/server/services/removebrokers/removal.engine.ts` — Removal automation:
+  - `submitAutomatedRemoval(broker, personalInfo)` — API-based removal where available
+  - `generateFormPayload(broker, personalInfo)` — prepare form data for manual submission
+  - `sendRemovalEmail(broker, personalInfo)` — email-based removal request
+  - `trackRemovalStatus(broker, requestId)` — poll broker for status updates
+
+steps:
+1. Create `web/src/server/api/routers/removebrokers.ts`.
+2. Define Zod schemas:
+   - `createRequestSchema`: `brokerId: z.string().uuid()`, `personalInfo: z.object({ fullName: z.string(), email: z.string().optional(), phone: z.string().optional(), address: z.string().optional(), dob: z.string().optional() })`
+   - `scanSchema`: `brokerId: z.string().uuid().optional()` (if omitted, scan all)
+3. Implement router procedures:
+   - Broker registry listing (public or protected)
+   - Removal request CRUD
+   - Listing scan and results
+   - Stats aggregation
+4. Create `web/src/server/services/removebrokers.service.ts`:
+   - Port from `services/removebrokers/src/`
+   - Implement request lifecycle: PENDING → SUBMITTED → IN_PROGRESS → COMPLETED/FAILED
+5. Create broker registry:
+   - Hardcode initial list of 20-50 major data brokers
+   - Include removal instructions and URLs
+   - Allow admin updates via API (future enhancement)
+6. Create removal engine:
+   - `submitAutomatedRemoval`: call broker API if available (rare)
+   - `generateFormPayload`: create structured data for form filling
+   - `sendRemovalEmail`: use notification service (task 14) to send removal request email
+   - `trackRemovalStatus`: placeholder for polling logic
+7. Implement listing scanner:
+   - Search broker websites for user's name, email, phone
+   - Use web scraping (Cheerio, Playwright) where APIs are unavailable
+   - Store found listings in `BrokerListing` table
+8. Implement scheduler integration:
+   - Pending removals should be picked up by background job (task 22)
+   - Retries for failed removals with exponential backoff
+9. Wire router into `web/src/server/api/root.ts`.
+10. Write unit tests with mocked broker interactions.
+
+steps:
+- Unit: `createRemovalRequest` creates record with PENDING status
+- Unit: `processRemovals` advances eligible requests to SUBMITTED
+- Unit: `scanForListings` creates BrokerListing records for found data
+- Unit: Broker registry returns correct metadata for known brokers
+- Integration: tRPC procedures enforce subscription scoping
+
+acceptance_criteria:
+- [ ] Broker registry lists all supported data brokers with removal methods
+- [ ] Removal requests can be created per broker with personal info
+- [ ] Request status tracks lifecycle from PENDING to COMPLETED/FAILED
+- [ ] Listings scanner finds user's data on broker sites
+- [ ] Automated removals use APIs where available; manual methods generate instructions
+- [ ] Failed removals are retried with exponential backoff
+- [ ] Stats endpoint shows removal progress (total, completed, pending, failed)
+
+validation:
+- List brokers and verify registry data
+- Create a removal request and verify DB record
+- Simulate a scan and verify listing records created
+- Run `cd web && pnpm test` for RemoveBrokers unit tests
+
+notes:
+- Reference legacy: `services/removebrokers/src/`, `packages/api/src/routes/removebrokers.routes.ts`
+- Data broker removal is often a manual process. The automation layer should handle the easy cases and provide clear instructions for manual cases.
+- Web scraping broker sites may violate Terms of Service. Use public APIs where available and document legal considerations.
+- Personal info submitted for removal should be handled carefully. Do not log raw PII.
+- The removal engine should be designed as a plugin system: each broker can have its own adapter for API, form, or email removal.
+- Consider integrating with a third-party service (e.g., DeleteMe, Optery) for broader broker coverage if building individual adapters is impractical.