shortcommings

tasks/core-services-implementation/04-darkwatch-attack-surface.md

@@ -0,0 +1,75 @@
+# 04. SecurityTrails, Censys, and Shodan API Integrations
+
+meta:
+  id: core-services-04
+  feature: core-services-implementation
+  priority: P1
+  depends_on: [core-services-03]
+  tags: [darkwatch, securitytrails, censys, shodan, attack-surface, api-integration]
+
+objective:
+- Integrate SecurityTrails, Censys, and Shodan APIs into the DarkWatch scan engine to monitor domain/IP attack surface exposure, complementing HIBP's breach monitoring.
+
+deliverables:
+- SecurityTrails client for DNS/WHOIS monitoring and subdomain enumeration
+- Censys client for internet-wide host scanning and certificate transparency
+- Shodan client for IoT/device exposure and Tor exit node monitoring
+- Unified exposure normalization from all three sources
+- Cost-aware scanning (respect rate limits, cache aggressively)
+
+steps:
+1. Sign up for API keys:
+   - SecurityTrails: https://securitytrails.com (free: 50 req/mo, Pro: $49/mo)
+   - Censys: https://censys.io (free: 250 req/mo, Pro: $79/mo)
+   - Shodan: https://shodan.io (free: 1,250 results/mo, Small Biz: $299/mo)
+2. Add `SECURITYTRAILS_API_KEY`, `CENSYS_API_ID`, `CENSYS_API_SECRET`, `SHODAN_API_KEY` to `.env.example`
+3. Create `darkwatch/securitytrails.client.ts`:
+   - `getDomainInfo(domain)` — WHOIS, DNS records, subdomains
+   - `getSubdomains(domain)` — enumerate all subdomains
+   - `getHistory(domain)` — historical DNS changes
+4. Create `darkwatch/censys.client.ts`:
+   - `searchHosts(query)` — find exposed hosts by IP/domain
+   - `getCertificates(domain)` — certificate transparency logs
+   - `viewHost(ip)` — detailed host fingerprinting
+5. Create `darkwatch/shodan.client.ts`:
+   - `search(query)` — search exposed devices and services
+   - `host(ip)` — detailed host information
+   - `count(query)` — result counts for monitoring
+6. Implement unified `processScanResult(source, result)` that normalizes all API responses to internal exposure schema
+7. Map exposure types:
+   - SecurityTrails: subdomain exposure, DNS misconfiguration, domain hijacking risk
+   - Censys: exposed services, outdated TLS, certificate issues
+   - Shodan: open ports, default credentials, IoT exposure, Tor association
+8. Add tier-aware scan limits: Shield = HIBP only, Guard+ = all sources
+9. Implement intelligent caching: cache SecurityTrails DNS data for 24h, Censys/Shodan for 7d
+10. Add cost-per-scan tracking in database for billing/usage analytics
+
+tests:
+- Unit: Mock all three API responses, verify normalization and exposure creation
+- Integration: Test each client against real APIs using low-risk test queries
+- E2E: Add domain to watchlist → trigger scan → verify exposures from all three sources
+
+acceptance_criteria:
+- [ ] SecurityTrails client queries real API and returns parsed domain/subdomain data
+- [ ] Censys client queries real API and returns host/certificate information
+- [ ] Shodan client queries real API and returns device/service exposure data
+- [ ] Each client respects rate limits (SecurityTrails: 10 req/sec, Censys: 200 req/min, Shodan: 5 req/sec)
+- [ ] Circuit breakers open after 3 failures and reset after 60 seconds for each source
+- [ ] Exposure records are normalized regardless of source (consistent schema)
+- [ ] Alerts are generated for critical findings (open admin panels, exposed databases, certificate expiry)
+- [ ] Cache hit reduces API calls — verify Redis stores and returns cached data
+- [ ] Cost tracking records API usage per scan for later billing optimization
+- [ ] Free tier users only get HIBP; paid tiers unlock SecurityTrails, Censys, Shodan
+
+validation:
+- Run `vitest run darkwatch.test.ts` — all tests pass
+- Manual: Query `example.com` across all three APIs, verify meaningful results returned
+- Check Redis: Cached responses reduce subsequent API calls
+- Monitor cost: API call counts tracked in database
+
+notes:
+- SecurityTrails is most useful for domain monitoring; Censys/Shodan for IP/host exposure
+- Shodan's dark web relevance is limited — it sees Tor exit nodes, not .onion content. Consider DarkOwl ($40K+/yr) for deep dark web later
+- The free tiers are sufficient for development but production needs paid plans ($500–$1,000/mo combined)
+- Focus on actionable findings: exposed RDP, default credentials, certificate expiry — not just raw port scans
+- The existing scan engine in `darkwatch.service.ts` already routes by watchlist item type — wire in new clients there