# 04. SecurityTrails, Censys, and Shodan API Integrations

meta:
  id: core-services-04
  feature: core-services-implementation
  priority: P1
  depends_on: [core-services-03]
  tags: [darkwatch, securitytrails, censys, shodan, attack-surface, api-integration]

objective:
- Integrate SecurityTrails, Censys, and Shodan APIs into the DarkWatch scan engine to monitor domain/IP attack surface exposure, complementing HIBP's breach monitoring.

deliverables:
- SecurityTrails client for DNS/WHOIS monitoring and subdomain enumeration
- Censys client for internet-wide host scanning and certificate transparency
- Shodan client for IoT/device exposure and Tor exit node monitoring
- Unified exposure normalization from all three sources
- Cost-aware scanning (respect rate limits, cache aggressively)

steps:
1. Sign up for API keys:
   - SecurityTrails: https://securitytrails.com (free: 50 req/mo, Pro: $49/mo)
   - Censys: https://censys.io (free: 250 req/mo, Pro: $79/mo)
   - Shodan: https://shodan.io (free: 1,250 results/mo, Small Biz: $299/mo)
2. Add `SECURITYTRAILS_API_KEY`, `CENSYS_API_ID`, `CENSYS_API_SECRET`, `SHODAN_API_KEY` to `.env.example`
3. Create `darkwatch/securitytrails.client.ts`:
   - `getDomainInfo(domain)` — WHOIS, DNS records, subdomains
   - `getSubdomains(domain)` — enumerate all subdomains
   - `getHistory(domain)` — historical DNS changes
4. Create `darkwatch/censys.client.ts`:
   - `searchHosts(query)` — find exposed hosts by IP/domain
   - `getCertificates(domain)` — certificate transparency logs
   - `viewHost(ip)` — detailed host fingerprinting
5. Create `darkwatch/shodan.client.ts`:
   - `search(query)` — search exposed devices and services
   - `host(ip)` — detailed host information
   - `count(query)` — result counts for monitoring
6. Implement unified `processScanResult(source, result)` that normalizes all API responses to internal exposure schema
7. Map exposure types:
   - SecurityTrails: subdomain exposure, DNS misconfiguration, domain hijacking risk
   - Censys: exposed services, outdated TLS, certificate issues
   - Shodan: open ports, default credentials, IoT exposure, Tor association
8. Add tier-aware scan limits: Shield = HIBP only, Guard+ = all sources
9. Implement intelligent caching: cache SecurityTrails DNS data for 24h, Censys/Shodan for 7d
10. Add cost-per-scan tracking in database for billing/usage analytics

tests:
- Unit: Mock all three API responses, verify normalization and exposure creation
- Integration: Test each client against real APIs using low-risk test queries
- E2E: Add domain to watchlist → trigger scan → verify exposures from all three sources

acceptance_criteria:
- [ ] SecurityTrails client queries real API and returns parsed domain/subdomain data
- [ ] Censys client queries real API and returns host/certificate information
- [ ] Shodan client queries real API and returns device/service exposure data
- [ ] Each client respects rate limits (SecurityTrails: 10 req/sec, Censys: 200 req/min, Shodan: 5 req/sec)
- [ ] Circuit breakers open after 3 failures and reset after 60 seconds for each source
- [ ] Exposure records are normalized regardless of source (consistent schema)
- [ ] Alerts are generated for critical findings (open admin panels, exposed databases, certificate expiry)
- [ ] Cache hit reduces API calls — verify Redis stores and returns cached data
- [ ] Cost tracking records API usage per scan for later billing optimization
- [ ] Free tier users only get HIBP; paid tiers unlock SecurityTrails, Censys, Shodan

validation:
- Run `vitest run darkwatch.test.ts` — all tests pass
- Manual: Query `example.com` across all three APIs, verify meaningful results returned
- Check Redis: Cached responses reduce subsequent API calls
- Monitor cost: API call counts tracked in database

notes:
- SecurityTrails is most useful for domain monitoring; Censys/Shodan for IP/host exposure
- Shodan's dark web relevance is limited — it sees Tor exit nodes, not .onion content. Consider DarkOwl ($40K+/yr) for deep dark web later
- The free tiers are sufficient for development but production needs paid plans ($500–$1,000/mo combined)
- Focus on actionable findings: exposed RDP, default credentials, certificate expiry — not just raw port scans
- The existing scan engine in `darkwatch.service.ts` already routes by watchlist item type — wire in new clients there