diff --git a/README.md b/README.md new file mode 100644 index 0000000..808dcea --- /dev/null +++ b/README.md @@ -0,0 +1,309 @@ +# ShieldAI + +**Multi-layered consumer identity protection against predatory AI-driven scams.** + +ShieldAI combines three detection engines — voice cloning detection, dark web monitoring, and real-time spam classification — to give consumers proactive defense against modern identity fraud. + +--- + +## The Pitch + +Scammers are weaponizing AI at scale: voice clones that sound exactly like your family, hyper-personalized phishing messages that bypass filters, and synthetic identities that exploit stolen data within hours of a breach. Legacy credit monitoring is reactive — it tells you after the damage is done. + +ShieldAI flips the model. We detect the scam _as it happens_: + +- **VoicePrint** analyzes inbound calls in real time to flag synthetic AI-generated voices before you're socially engineered. +- **DarkWatch** continuously monitors dark web forums, breach databases, and data broker caches — notifying you the moment your credentials, phone, or SSN surface. +- **SpamShield** intercepts and classifies spam calls and SMS at the network level, blocking threats before they reach your phone. + +Backed by ML models (ECAPA-TDNN, BERT) and a real-time alert pipeline, ShieldAI gives consumers enterprise-grade threat detection for their personal life. + +--- + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────┐ +│ Clients │ +│ Mobile (Expo/RN) │ Web (SolidJS) │ Browser Extension │ +└──────────┬──────────────────────────────┬───────────────┘ + │ REST + WebSocket │ + ▼ ▼ +┌──────────────────────┐ ┌──────────────────────────────┐ +│ API Gateway │ │ WebSocket Alert Server │ +│ (Fastify 5) │ │ (Real-time push) │ +└──────┬───────┬───────┘ └──────────────────────────────┘ + │ │ + ▼ ▼ +┌──────────┐ ┌─────────────────────────────────────────────┐ +│ Auth │ │ Microservices │ +│(NextAuth)│ │ VoicePrint │ DarkWatch │ SpamShield │ +└──────────┘ │ HomeTitle │ RemoveBrokers │ + └──────────┬──────────┬───────────────────────┘ + │ │ + ┌────────▼──────────▼────────┐ + │ Background Workers │ + │ (BullMQ + Redis) │ + └────────┬───────────────────┘ + │ + ┌────────▼────────┐ + │ PostgreSQL │ + │ + Redis │ + └─────────────────┘ +``` + +--- + +## Features & Implementation Status + +| Feature | Service |Status | Notes | +|---------|---------|--------|-------| +| Voice enrollment & profile management | VoicePrint | ✅ Done | Register family voice profiles | +| Audio preprocessing (VAD, noise reduction) | VoicePrint | ✅ Done | WebRTC VAD + RNNoise | +| Synthetic voice detection (ECAPA-TDNN) | VoicePrint | ✅ Done | FAISS vector index for matching | +| Real-time streaming audio analysis | VoicePrint | ✅ Done | WebSocket-based | +| Batch audio analysis | VoicePrint | ✅ Done | Configurable confidence thresholds per tier | +| HIBP breach checking | DarkWatch | ✅ Done | Email + password breach lookup | +| Dark web multi-source scanning | DarkWatch | ✅ Done | HIBP, SecurityTrails, Censys, Shodan, forums | +| Watch list management | DarkWatch | ✅ Done | Emails, phones, SSN (hashed) | +| Scheduled + real-time scanning | DarkWatch | ✅ Done | Tier-based frequency | +| Fuzzy matching engine | DarkWatch | ✅ Done | Levenshtein + exact matching | +| Severity-scored alert pipeline | DarkWatch | ✅ Done | Dedup pipeline | +| PDF report generation | DarkWatch | ✅ Done | Handlebars + PDFKit | +| Number reputation (Hiya/Truecaller) | SpamShield | ✅ Done | Circuit breaker pattern | +| SMS classification (BERT) | SpamShield | ✅ Done | ML-based spam detection | +| Call analysis rule engine | SpamShield | ✅ Done | Multi-layer scoring | +| Real-time carrier interception | SpamShield | ⏳ In Progress | Twilio/Plivo integration | +| Real-time WebSocket alerts | SpamShield | ✅ Done | Alert broadcasting | +| User feedback loop (FP/FN) | SpamShield | ✅ Done | Metadata validation | +| Phone validation (E.164) | SpamShield | ✅ Done | Normalization | +| Audit logging | SpamShield | ✅ Done | All decisions logged | +| SpamShield rate limiting | SpamShield | ⏳ In Progress | Per-endpoint + global | +| SpamShield route optimization | SpamShield | ⏳ In Progress | Route consolidation | +| Feature flags | All | ✅ Done | Env-variable toggles | +| Property record matching | HomeTitle | ✅ Done | Fuzzy string matching | +| Change detection (ownership, liens) | HomeTitle | ✅ Done | County deed scanning | +| Watchlist management | HomeTitle | ✅ Done | | +| Scheduled county deed scanning | HomeTitle | ✅ Done | | +| Alert pipeline | HomeTitle | ✅ Done | Severity classification | +| Data broker removal requests | RemoveBrokers | ✅ Done | | +| Broker API integration | RemoveBrokers | ✅ Done | With caching | +| User auth (JWT, RBAC) | Shared | ✅ Done | NextAuth.js | +| Family group management | Shared | ✅ Done | | +| Stripe subscriptions & billing | Shared | ✅ Done | Tier-based feature gating | +| Email (Resend) | Shared | ✅ Done | Transactional + marketing | +| Push notifications (FCM/APNs) | Shared | ✅ Done | Android + iOS | +| SMS (Twilio) | Shared | ✅ Done | | +| Mixpanel analytics (30+ events) | Shared | ✅ Done | KPI tracking | +| Datadog APM + Sentry | Shared | ✅ Done | Full observability | +| Cross-service event correlation | Shared | ✅ Done | Alert correlation engine | +| Browser extension (MV3) | Extension | ✅ Done | Phishing detection | +| Mobile app (Expo RN) | Mobile | ✅ Done | iOS + Android | +| Shared UI component library | Shared UI | ✅ Done | SolidJS | +| CI/CD pipelines | DevOps | ✅ Done | GitHub Actions | +| Terraform infrastructure | DevOps | ✅ Done | AWS ECS, RDS, ElastiCache | +| Load testing (k6) | DevOps | ✅ Done | VoicePrint + DarkWatch | +| Docker + Compose | DevOps | ✅ Done | Dev + prod configs | +| Integration tests | QA | ⏳ In Progress | Coverage expanding | +| Rate limit tests | QA | ⏳ In Progress | | + +--- + +## Tech Stack + +| Layer | Technology | +|-------|-----------| +| **Language** | TypeScript (Node.js ≥20) | +| **API** | Fastify 5 (CORS, Helmet, rate-limit, Swagger, multipart) | +| **Frontend** | SolidJS + Vite | +| **Mobile** | React Native / Expo SDK 51 | +| **Extension** | Chrome Manifest V3 | +| **Databases** | PostgreSQL 15/16 (Prisma ORM) + Turso/SQLite (Drizzle) | +| **Cache / Queue** | Redis + BullMQ | +| **Auth** | NextAuth.js + JWT | +| **Billing** | Stripe | +| **Email** | Resend | +| **Push** | Firebase Cloud Messaging + APNs | +| **SMS** | Twilio | +| **Analytics** | Mixpanel / Segment | +| **Monitoring** | Datadog APM + Sentry | +| **ML Models** | ECAPA-TDNN (voice), BERT (SMS), FAISS (vector index) | +| **Infrastructure** | Terraform (AWS ECS Fargate, RDS, ElastiCache, S3, ALB) | +| **CI/CD** | GitHub Actions | +| **Monorepo** | pnpm workspaces + Turborepo | +| **Testing** | Vitest, Jest, k6 | + +--- + +## Getting Started + +### Prerequisites + +- Node.js >= 20.0.0 +- pnpm >= 9.0.0 +- Docker & Docker Compose + +### Setup + +```bash +# Install all dependencies +pnpm install + +# Start local infrastructure (Postgres, Redis, Mailhog) +docker compose up -d + +# Copy environment variables +cp .env.example .env + +# Run database migrations +pnpm db:migrate + +# Start all development servers +pnpm dev +``` + +This launches the API server, all microservices, and the web frontend concurrently via Turborepo. + +--- + +## Building + +```bash +# Build all packages and services +pnpm build + +# Build individual Docker images +docker build -f packages/api/Dockerfile -t shieldai-api . +docker build -f services/spamshield/Dockerfile -t shieldai-spamshield . +docker build -f services/darkwatch/Dockerfile -t shieldai-darkwatch . +docker build -f services/voiceprint/Dockerfile -t shieldai-voiceprint . +``` + +--- + +## Testing + +```bash +# Run all tests +pnpm test + +# With coverage +pnpm test:coverage + +# Individual service tests +pnpm test --filter @shieldai/spamshield +pnpm test --filter @shieldai/darkwatch +pnpm test --filter @shieldai/voiceprint +pnpm test --filter @shieldai/hometitle + +# Integration & E2E +cd packages/integration-tests && pnpm test +cd packages/integration-tests && pnpm test:e2e + +# Load tests (requires k6) +cd scripts/load-test && ./run-all.sh +``` + +--- + +## Production + +### Docker Compose + +```bash +docker compose -f docker-compose.prod.yml up -d +``` + +### CI/CD Pipeline (GitHub Actions) + +| Event | Deploy To | +|-------|-----------| +| Push to `main` | Staging | +| GitHub Release created | Production | + +Pipeline stages: `lint` → `typecheck` → `test` → `Docker build` → `push to GHCR` → `Terraform apply` → `ECS deploy` → `health check` → auto-rollback on failure. + +### Infrastructure + +All infrastructure lives in `infra/` and is managed with Terraform: + +- **Compute**: AWS ECS Fargate (API + services + workers) +- **Database**: RDS PostgreSQL 15/16 +- **Cache**: ElastiCache Redis +- **Storage**: S3 (reports, audio samples) +- **Networking**: VPC, ALB, security groups +- **Observability**: CloudWatch + Datadog +- **Secrets**: AWS Secrets Manager + +See `infra/README.md` and `infra/ROLLBACK.md` for detailed operational runbooks. + +--- + +## Project Structure + +``` +shieldai/ +├── packages/ # Shared libraries (20 packages) +│ ├── api/ # Fastify API server +│ ├── core/ # Core shared logic +│ ├── db/ # Prisma schemas (v6) +│ ├── shared-db/ # Prisma schemas (v5) +│ ├── shared-auth/ # NextAuth.js +│ ├── shared-billing/ # Stripe subscriptions +│ ├── shared-notifications/ # Email, Push, SMS +│ ├── shared-analytics/ # Mixpanel/Segment +│ ├── shared-ui/ # SolidJS components +│ ├── shared-utils/ # Utilities +│ ├── types/ # Shared TypeScript types +│ ├── mobile/ # React Native / Expo app +│ ├── extension/ # Browser extension (MV3) +│ ├── jobs/ # BullMQ workers +│ ├── monitoring/ # Datadog + Sentry +│ ├── report/ # PDF generation +│ ├── correlation/ # Event correlation +│ ├── mobile-api-client/ # RN API client +│ └── integration-tests/ # E2E tests +├── services/ # Microservices (5) +│ ├── voiceprint/ # Voice cloning detection +│ ├── darkwatch/ # Dark web monitoring +│ ├── spamshield/ # Spam call/SMS blocking +│ ├── hometitle/ # Home title fraud +│ └── removebrokers/ # Data broker removal +├── infra/ # Terraform (AWS) +├── docs/ # Documentation +├── plans/ # Product & technical plans +├── scripts/ # Utility scripts +├── load-tests/ # k6 load test scripts +├── assets/ # Ad creative assets +└── server/ # Legacy WebSocket server +``` + +--- + +## Documentation + +| Document | Location | +|----------|----------| +| Product Plan | `plans/SHIELDAI-product-plan.md` | +| Technical Architecture | `plans/SHIELDAI-technical-architecture.md` | +| Infrastructure | `infra/README.md` | +| Rollback Runbook | `infra/ROLLBACK.md` | +| Stripe Integration | `docs/STRIPE_INTEGRATION.md` | +| Push Notifications | `docs/PUSH_NOTIFICATIONS_SETUP.md` | +| Mobile Push Integration | `docs/MOBILE_PUSH_INTEGRATION.md` | +| Mixpanel Analytics | `docs/MIXPANEL_ANALYTICS.md` | +| Code Review Workflow | `shieldai-workflow.md` | + +--- + +## Subscription Tiers + +| Feature | Basic (Free) | Plus ($9.99/mo) | Premium ($24.99/mo) | +|---------|:------------:|:----------------:|:-------------------:| +| Dark web scans | Limited | Unlimited | Unlimited | +| Spam detection | Basic | AI-powered | AI-powered | +| Voice cloning detection | — | Family | Family | +| SSN monitoring | — | — | ✅ | +| Home title protection | — | — | ✅ | +| Real-time blocking | — | — | ✅ | +| 24/7 support | — | — | ✅ | diff --git a/shieldai-workflow.md b/shieldai-workflow.md deleted file mode 100644 index 19cd8c7..0000000 --- a/shieldai-workflow.md +++ /dev/null @@ -1,83 +0,0 @@ -# ShieldAI Code Review Workflow - -## Current State (as of May 2, 2026) - -### PR Backlog Status -- **Open PRs**: 0 (pending commits pushed to master) -- **Pending commits**: 1 commit pushed (FRE-4604) — remaining 6 were previously pushed -- **Last review cycle**: FRE-4500, FRE-4499, FRE-4612 (security findings — all done) -- **Branch protection**: Configured (see `branch-protection-rules.yaml`) -- **PR template**: Configured (`.gitea/pull_request_templates/default.md`) - -### Resolved Bottlenecks -1. ✅ PR-based workflow established with PR template -2. ✅ Branch protection rules documented and configured -3. ✅ Code review checklist integrated into PR template -4. ✅ Security review findings integrated (FRE-4499, FRE-4500, FRE-4612 all done) - -## PR Process - -1. **Feature branch creation** from `gt/master` -2. **Development commits** with conventional commit format (include issue ID: `FRE-XXXX: description`) -3. **PR creation** against `gt/master` -4. **Required reviews**: - - Code Reviewer — all PRs - - Security Reviewer — for security-sensitive changes -5. **CI checks** pass (lint, typecheck, test) -6. **Merge** via squash or rebase - -### Code Review Checklist - -- [ ] Security impact assessment -- [ ] Test coverage verification -- [ ] Type checking (TypeScript) -- [ ] Linting compliance -- [ ] Documentation updates -- [ ] Breaking changes documented -- [ ] Backward compatibility verified - -### Branch Protection Rules - -See `branch-protection-rules.yaml` for the full configuration. Summary: - -- **Protected branch**: `gt/master` -- **Required reviews**: 1 approved review before merge -- **Required status checks**: lint, typecheck, test -- **Enforce admins**: false (admins can bypass during emergencies) -- **Allow force pushes**: true (for recovery scenarios) - -## Review Assignment Policy - -| Change Type | Required Reviewers | -|-------------|-------------------| -| General code | Code Reviewer | -| Security-critical | Code Reviewer + Security Reviewer | -| API contracts | Code Reviewer + CTO | -| Database schema | Code Reviewer + Senior Engineer | - -## Review Pipeline - -``` -Engineer implements → marks in_review → Security Reviewer reviews → Code Reviewer reviews → Done -``` - -## Metrics to Track - -- PR cycle time (creation to merge) -- Review turnaround time -- PR size (lines changed) -- Review comments per PR -- Merge conflict frequency - -## Contribution Guidelines - -1. Always create a feature branch from `gt/master` -2. Use conventional commit format: `type(scope): description (FRE-XXXX)` -3. Include tests for new functionality -4. Update documentation for API changes -5. Run lint and typecheck before pushing -6. Create PR with filled template before requesting review -7. Address all review comments before merge - ---- -*Updated from FRE-4556 audit, implemented in FRE-4661*