76 lines
4.3 KiB
Markdown
76 lines
4.3 KiB
Markdown
# 04. SecurityTrails, Censys, and Shodan API Integrations
|
||
|
||
meta:
|
||
id: core-services-04
|
||
feature: core-services-implementation
|
||
priority: P1
|
||
depends_on: [core-services-03]
|
||
tags: [darkwatch, securitytrails, censys, shodan, attack-surface, api-integration]
|
||
|
||
objective:
|
||
- Integrate SecurityTrails, Censys, and Shodan APIs into the DarkWatch scan engine to monitor domain/IP attack surface exposure, complementing HIBP's breach monitoring.
|
||
|
||
deliverables:
|
||
- SecurityTrails client for DNS/WHOIS monitoring and subdomain enumeration
|
||
- Censys client for internet-wide host scanning and certificate transparency
|
||
- Shodan client for IoT/device exposure and Tor exit node monitoring
|
||
- Unified exposure normalization from all three sources
|
||
- Cost-aware scanning (respect rate limits, cache aggressively)
|
||
|
||
steps:
|
||
1. Sign up for API keys:
|
||
- SecurityTrails: https://securitytrails.com (free: 50 req/mo, Pro: $49/mo)
|
||
- Censys: https://censys.io (free: 250 req/mo, Pro: $79/mo)
|
||
- Shodan: https://shodan.io (free: 1,250 results/mo, Small Biz: $299/mo)
|
||
2. Add `SECURITYTRAILS_API_KEY`, `CENSYS_API_ID`, `CENSYS_API_SECRET`, `SHODAN_API_KEY` to `.env.example`
|
||
3. Create `darkwatch/securitytrails.client.ts`:
|
||
- `getDomainInfo(domain)` — WHOIS, DNS records, subdomains
|
||
- `getSubdomains(domain)` — enumerate all subdomains
|
||
- `getHistory(domain)` — historical DNS changes
|
||
4. Create `darkwatch/censys.client.ts`:
|
||
- `searchHosts(query)` — find exposed hosts by IP/domain
|
||
- `getCertificates(domain)` — certificate transparency logs
|
||
- `viewHost(ip)` — detailed host fingerprinting
|
||
5. Create `darkwatch/shodan.client.ts`:
|
||
- `search(query)` — search exposed devices and services
|
||
- `host(ip)` — detailed host information
|
||
- `count(query)` — result counts for monitoring
|
||
6. Implement unified `processScanResult(source, result)` that normalizes all API responses to internal exposure schema
|
||
7. Map exposure types:
|
||
- SecurityTrails: subdomain exposure, DNS misconfiguration, domain hijacking risk
|
||
- Censys: exposed services, outdated TLS, certificate issues
|
||
- Shodan: open ports, default credentials, IoT exposure, Tor association
|
||
8. Add tier-aware scan limits: Shield = HIBP only, Guard+ = all sources
|
||
9. Implement intelligent caching: cache SecurityTrails DNS data for 24h, Censys/Shodan for 7d
|
||
10. Add cost-per-scan tracking in database for billing/usage analytics
|
||
|
||
tests:
|
||
- Unit: Mock all three API responses, verify normalization and exposure creation
|
||
- Integration: Test each client against real APIs using low-risk test queries
|
||
- E2E: Add domain to watchlist → trigger scan → verify exposures from all three sources
|
||
|
||
acceptance_criteria:
|
||
- [ ] SecurityTrails client queries real API and returns parsed domain/subdomain data
|
||
- [ ] Censys client queries real API and returns host/certificate information
|
||
- [ ] Shodan client queries real API and returns device/service exposure data
|
||
- [ ] Each client respects rate limits (SecurityTrails: 10 req/sec, Censys: 200 req/min, Shodan: 5 req/sec)
|
||
- [ ] Circuit breakers open after 3 failures and reset after 60 seconds for each source
|
||
- [ ] Exposure records are normalized regardless of source (consistent schema)
|
||
- [ ] Alerts are generated for critical findings (open admin panels, exposed databases, certificate expiry)
|
||
- [ ] Cache hit reduces API calls — verify Redis stores and returns cached data
|
||
- [ ] Cost tracking records API usage per scan for later billing optimization
|
||
- [ ] Free tier users only get HIBP; paid tiers unlock SecurityTrails, Censys, Shodan
|
||
|
||
validation:
|
||
- Run `vitest run darkwatch.test.ts` — all tests pass
|
||
- Manual: Query `example.com` across all three APIs, verify meaningful results returned
|
||
- Check Redis: Cached responses reduce subsequent API calls
|
||
- Monitor cost: API call counts tracked in database
|
||
|
||
notes:
|
||
- SecurityTrails is most useful for domain monitoring; Censys/Shodan for IP/host exposure
|
||
- Shodan's dark web relevance is limited — it sees Tor exit nodes, not .onion content. Consider DarkOwl ($40K+/yr) for deep dark web later
|
||
- The free tiers are sufficient for development but production needs paid plans ($500–$1,000/mo combined)
|
||
- Focus on actionable findings: exposed RDP, default credentials, certificate expiry — not just raw port scans
|
||
- The existing scan engine in `darkwatch.service.ts` already routes by watchlist item type — wire in new clients there
|