Kordant/tasks/core-services-implementation/04-darkwatch-attack-surface.md

04. SecurityTrails, Censys, and Shodan API Integrations

meta: id: core-services-04 feature: core-services-implementation priority: P1 depends_on: [core-services-03] tags: [darkwatch, securitytrails, censys, shodan, attack-surface, api-integration]

objective:

• Integrate SecurityTrails, Censys, and Shodan APIs into the DarkWatch scan engine to monitor domain/IP attack surface exposure, complementing HIBP's breach monitoring.

deliverables:

• SecurityTrails client for DNS/WHOIS monitoring and subdomain enumeration
• Censys client for internet-wide host scanning and certificate transparency
• Shodan client for IoT/device exposure and Tor exit node monitoring
• Unified exposure normalization from all three sources
• Cost-aware scanning (respect rate limits, cache aggressively)

steps:

1. Sign up for API keys:
   • SecurityTrails: https://securitytrails.com (free: 50 req/mo, Pro: $49/mo)
   • Censys: https://censys.io (free: 250 req/mo, Pro: $79/mo)
   • Shodan: https://shodan.io (free: 1,250 results/mo, Small Biz: $299/mo)
2. Add SECURITYTRAILS_API_KEY, CENSYS_API_ID, CENSYS_API_SECRET, SHODAN_API_KEY to .env.example
3. Create darkwatch/securitytrails.client.ts:
   • getDomainInfo(domain) — WHOIS, DNS records, subdomains
   • getSubdomains(domain) — enumerate all subdomains
   • getHistory(domain) — historical DNS changes
4. Create darkwatch/censys.client.ts:
   • searchHosts(query) — find exposed hosts by IP/domain
   • getCertificates(domain) — certificate transparency logs
   • viewHost(ip) — detailed host fingerprinting
5. Create darkwatch/shodan.client.ts:
   • search(query) — search exposed devices and services
   • host(ip) — detailed host information
   • count(query) — result counts for monitoring
6. Implement unified processScanResult(source, result) that normalizes all API responses to internal exposure schema
7. Map exposure types:
   • SecurityTrails: subdomain exposure, DNS misconfiguration, domain hijacking risk
   • Censys: exposed services, outdated TLS, certificate issues
   • Shodan: open ports, default credentials, IoT exposure, Tor association
8. Add tier-aware scan limits: Shield = HIBP only, Guard+ = all sources
9. Implement intelligent caching: cache SecurityTrails DNS data for 24h, Censys/Shodan for 7d
10. Add cost-per-scan tracking in database for billing/usage analytics

tests:

• Unit: Mock all three API responses, verify normalization and exposure creation
• Integration: Test each client against real APIs using low-risk test queries
• E2E: Add domain to watchlist → trigger scan → verify exposures from all three sources

acceptance_criteria:

• SecurityTrails client queries real API and returns parsed domain/subdomain data
• Censys client queries real API and returns host/certificate information
• Shodan client queries real API and returns device/service exposure data
• Each client respects rate limits (SecurityTrails: 10 req/sec, Censys: 200 req/min, Shodan: 5 req/sec)
• Circuit breakers open after 3 failures and reset after 60 seconds for each source
• Exposure records are normalized regardless of source (consistent schema)
• Alerts are generated for critical findings (open admin panels, exposed databases, certificate expiry)
• Cache hit reduces API calls — verify Redis stores and returns cached data
• Cost tracking records API usage per scan for later billing optimization
• Free tier users only get HIBP; paid tiers unlock SecurityTrails, Censys, Shodan

validation:

• Run vitest run darkwatch.test.ts — all tests pass
• Manual: Query example.com across all three APIs, verify meaningful results returned
• Check Redis: Cached responses reduce subsequent API calls
• Monitor cost: API call counts tracked in database

notes:

• SecurityTrails is most useful for domain monitoring; Censys/Shodan for IP/host exposure
• Shodan's dark web relevance is limited — it sees Tor exit nodes, not .onion content. Consider DarkOwl ($40K+/yr) for deep dark web later
• The free tiers are sufficient for development but production needs paid plans ($500–$1,000/mo combined)
• Focus on actionable findings: exposed RDP, default credentials, certificate expiry — not just raw port scans
• The existing scan engine in darkwatch.service.ts already routes by watchlist item type — wire in new clients there