# 11. Application Metrics & Dashboards

meta:
  id: web-production-11
  feature: web-production
  priority: P2
  depends_on: []
  tags: [observability, metrics, production]

objective:
- Collect and visualize application metrics for performance monitoring and capacity planning

deliverables:
- Prometheus metrics endpoint
- Custom business metrics
- Grafana or Datadog dashboards
- Alerting on metric thresholds

steps:
1. Add metrics collection:
   - Install prom-client for Node.js metrics
   - Create web/src/server/lib/metrics.ts
   - Expose /metrics endpoint for Prometheus scraping
2. Collect standard metrics:
   - HTTP request duration (histogram)
   - HTTP request count (counter, by status code, endpoint)
   - Active connections (gauge)
   - Memory usage (gauge)
   - Event loop lag (gauge)
3. Collect business metrics:
   - Signup rate (counter)
   - Login success/failure rate (counter)
   - Subscription conversions (counter)
   - DarkWatch scan completions (counter)
   - Alert generation rate (counter)
   - Average threat score (gauge)
4. Set up dashboards:
   - Grafana dashboard or Datadog dashboard
   - Request latency percentiles (p50, p95, p99)
   - Error rate over time
   - Business funnel (landing → signup → subscribe)
   - Infrastructure health (CPU, memory, DB connections)
5. Configure alerts:
   - p99 latency > 500ms for 5 minutes
   - Error rate > 1% for 2 minutes
   - Memory usage > 80% for 10 minutes
   - DB connection pool > 90% for 5 minutes

tests:
- Unit: Test metrics increment correctly
- Integration: Verify /metrics endpoint returns valid Prometheus format
- Dashboard: Confirm all panels show data

acceptance_criteria:
- /metrics endpoint serving valid Prometheus exposition format
- Request duration histogram with 0.1, 0.5, 1, 2, 5 second buckets
- Business metrics visible in dashboard
- Alert fires when p99 latency exceeds 500ms
- Dashboard refreshes every 10 seconds with live data
- Metrics retention for 30 days

validation:
- `curl /metrics` → valid Prometheus output
- Grafana dashboard shows request latency graph
- Trigger slow endpoint → alert fires within 5 minutes

notes:
- Prometheus + Grafana is open source and cost-effective
- Datadog is easier but more expensive
- Consider using Vercel Analytics if deployed on Vercel