67 lines
2.4 KiB
Markdown
67 lines
2.4 KiB
Markdown
# 09. Structured Logging & Log Aggregation
|
|
|
|
meta:
|
|
id: web-production-09
|
|
feature: web-production
|
|
priority: P2
|
|
depends_on: []
|
|
tags: [observability, logging, production]
|
|
|
|
objective:
|
|
- Replace ad-hoc logging with structured, aggregated logging for production debugging and auditing
|
|
|
|
deliverables:
|
|
- Structured logging library integration (Pino or Winston)
|
|
- Log aggregation pipeline (Datadog, Logtail, or CloudWatch)
|
|
- Request ID propagation across all logs
|
|
- Log rotation and retention policy
|
|
|
|
steps:
|
|
1. Add structured logging library:
|
|
- Install pino or winston in web/package.json
|
|
- Create web/src/server/lib/logger.ts with configured logger
|
|
- Replace all console.log/console.error with logger
|
|
2. Implement request context logging:
|
|
- Generate request ID for each incoming request
|
|
- Attach user ID, session ID to log context
|
|
- Propagate request ID through tRPC context
|
|
3. Configure log levels:
|
|
- ERROR: unhandled exceptions, auth failures, DB errors
|
|
- WARN: rate limit hits, slow queries, deprecated API usage
|
|
- INFO: requests, logins, signups, billing events
|
|
- DEBUG: query details, cache hits/misses (dev only)
|
|
4. Set up log aggregation:
|
|
- Configure log shipping to aggregation service
|
|
- Set up log parsing and indexing
|
|
- Create saved searches for common issues
|
|
5. Implement log rotation:
|
|
- 100MB max per file
|
|
- 7 days retention for production
|
|
- 30 days retention for audit logs
|
|
6. Add sensitive data redaction:
|
|
- Mask credit card numbers, SSNs, passwords in logs
|
|
- Redact JWT tokens (show only first 10 chars)
|
|
|
|
tests:
|
|
- Unit: Test logger outputs valid JSON
|
|
- Integration: Test request ID propagation
|
|
- Security: Verify no sensitive data in logs
|
|
|
|
acceptance_criteria:
|
|
- All logs output as structured JSON
|
|
- Request ID present on every log line for a given request
|
|
- Log aggregation service receiving logs in real-time
|
|
- Sensitive data redacted from all log output
|
|
- Log rotation preventing disk fill
|
|
- Searchable logs by user ID, request ID, endpoint
|
|
|
|
validation:
|
|
- Trigger error → log appears in aggregation with stack trace, request ID, user ID
|
|
- Search logs by request ID → all related logs returned
|
|
- Check log files → no credit card numbers, passwords, full JWTs
|
|
|
|
notes:
|
|
- Pino is fastest and recommended for Node.js
|
|
- Use pino-pretty for local development, JSON for production
|
|
- Consider OpenTelemetry for unified tracing + logging
|