update readme
This commit is contained in:
309
README.md
Normal file
309
README.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# ShieldAI
|
||||
|
||||
**Multi-layered consumer identity protection against predatory AI-driven scams.**
|
||||
|
||||
ShieldAI combines three detection engines — voice cloning detection, dark web monitoring, and real-time spam classification — to give consumers proactive defense against modern identity fraud.
|
||||
|
||||
---
|
||||
|
||||
## The Pitch
|
||||
|
||||
Scammers are weaponizing AI at scale: voice clones that sound exactly like your family, hyper-personalized phishing messages that bypass filters, and synthetic identities that exploit stolen data within hours of a breach. Legacy credit monitoring is reactive — it tells you after the damage is done.
|
||||
|
||||
ShieldAI flips the model. We detect the scam _as it happens_:
|
||||
|
||||
- **VoicePrint** analyzes inbound calls in real time to flag synthetic AI-generated voices before you're socially engineered.
|
||||
- **DarkWatch** continuously monitors dark web forums, breach databases, and data broker caches — notifying you the moment your credentials, phone, or SSN surface.
|
||||
- **SpamShield** intercepts and classifies spam calls and SMS at the network level, blocking threats before they reach your phone.
|
||||
|
||||
Backed by ML models (ECAPA-TDNN, BERT) and a real-time alert pipeline, ShieldAI gives consumers enterprise-grade threat detection for their personal life.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Clients │
|
||||
│ Mobile (Expo/RN) │ Web (SolidJS) │ Browser Extension │
|
||||
└──────────┬──────────────────────────────┬───────────────┘
|
||||
│ REST + WebSocket │
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌──────────────────────────────┐
|
||||
│ API Gateway │ │ WebSocket Alert Server │
|
||||
│ (Fastify 5) │ │ (Real-time push) │
|
||||
└──────┬───────┬───────┘ └──────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────┐ ┌─────────────────────────────────────────────┐
|
||||
│ Auth │ │ Microservices │
|
||||
│(NextAuth)│ │ VoicePrint │ DarkWatch │ SpamShield │
|
||||
└──────────┘ │ HomeTitle │ RemoveBrokers │
|
||||
└──────────┬──────────┬───────────────────────┘
|
||||
│ │
|
||||
┌────────▼──────────▼────────┐
|
||||
│ Background Workers │
|
||||
│ (BullMQ + Redis) │
|
||||
└────────┬───────────────────┘
|
||||
│
|
||||
┌────────▼────────┐
|
||||
│ PostgreSQL │
|
||||
│ + Redis │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Features & Implementation Status
|
||||
|
||||
| Feature | Service |Status | Notes |
|
||||
|---------|---------|--------|-------|
|
||||
| Voice enrollment & profile management | VoicePrint | ✅ Done | Register family voice profiles |
|
||||
| Audio preprocessing (VAD, noise reduction) | VoicePrint | ✅ Done | WebRTC VAD + RNNoise |
|
||||
| Synthetic voice detection (ECAPA-TDNN) | VoicePrint | ✅ Done | FAISS vector index for matching |
|
||||
| Real-time streaming audio analysis | VoicePrint | ✅ Done | WebSocket-based |
|
||||
| Batch audio analysis | VoicePrint | ✅ Done | Configurable confidence thresholds per tier |
|
||||
| HIBP breach checking | DarkWatch | ✅ Done | Email + password breach lookup |
|
||||
| Dark web multi-source scanning | DarkWatch | ✅ Done | HIBP, SecurityTrails, Censys, Shodan, forums |
|
||||
| Watch list management | DarkWatch | ✅ Done | Emails, phones, SSN (hashed) |
|
||||
| Scheduled + real-time scanning | DarkWatch | ✅ Done | Tier-based frequency |
|
||||
| Fuzzy matching engine | DarkWatch | ✅ Done | Levenshtein + exact matching |
|
||||
| Severity-scored alert pipeline | DarkWatch | ✅ Done | Dedup pipeline |
|
||||
| PDF report generation | DarkWatch | ✅ Done | Handlebars + PDFKit |
|
||||
| Number reputation (Hiya/Truecaller) | SpamShield | ✅ Done | Circuit breaker pattern |
|
||||
| SMS classification (BERT) | SpamShield | ✅ Done | ML-based spam detection |
|
||||
| Call analysis rule engine | SpamShield | ✅ Done | Multi-layer scoring |
|
||||
| Real-time carrier interception | SpamShield | ⏳ In Progress | Twilio/Plivo integration |
|
||||
| Real-time WebSocket alerts | SpamShield | ✅ Done | Alert broadcasting |
|
||||
| User feedback loop (FP/FN) | SpamShield | ✅ Done | Metadata validation |
|
||||
| Phone validation (E.164) | SpamShield | ✅ Done | Normalization |
|
||||
| Audit logging | SpamShield | ✅ Done | All decisions logged |
|
||||
| SpamShield rate limiting | SpamShield | ⏳ In Progress | Per-endpoint + global |
|
||||
| SpamShield route optimization | SpamShield | ⏳ In Progress | Route consolidation |
|
||||
| Feature flags | All | ✅ Done | Env-variable toggles |
|
||||
| Property record matching | HomeTitle | ✅ Done | Fuzzy string matching |
|
||||
| Change detection (ownership, liens) | HomeTitle | ✅ Done | County deed scanning |
|
||||
| Watchlist management | HomeTitle | ✅ Done | |
|
||||
| Scheduled county deed scanning | HomeTitle | ✅ Done | |
|
||||
| Alert pipeline | HomeTitle | ✅ Done | Severity classification |
|
||||
| Data broker removal requests | RemoveBrokers | ✅ Done | |
|
||||
| Broker API integration | RemoveBrokers | ✅ Done | With caching |
|
||||
| User auth (JWT, RBAC) | Shared | ✅ Done | NextAuth.js |
|
||||
| Family group management | Shared | ✅ Done | |
|
||||
| Stripe subscriptions & billing | Shared | ✅ Done | Tier-based feature gating |
|
||||
| Email (Resend) | Shared | ✅ Done | Transactional + marketing |
|
||||
| Push notifications (FCM/APNs) | Shared | ✅ Done | Android + iOS |
|
||||
| SMS (Twilio) | Shared | ✅ Done | |
|
||||
| Mixpanel analytics (30+ events) | Shared | ✅ Done | KPI tracking |
|
||||
| Datadog APM + Sentry | Shared | ✅ Done | Full observability |
|
||||
| Cross-service event correlation | Shared | ✅ Done | Alert correlation engine |
|
||||
| Browser extension (MV3) | Extension | ✅ Done | Phishing detection |
|
||||
| Mobile app (Expo RN) | Mobile | ✅ Done | iOS + Android |
|
||||
| Shared UI component library | Shared UI | ✅ Done | SolidJS |
|
||||
| CI/CD pipelines | DevOps | ✅ Done | GitHub Actions |
|
||||
| Terraform infrastructure | DevOps | ✅ Done | AWS ECS, RDS, ElastiCache |
|
||||
| Load testing (k6) | DevOps | ✅ Done | VoicePrint + DarkWatch |
|
||||
| Docker + Compose | DevOps | ✅ Done | Dev + prod configs |
|
||||
| Integration tests | QA | ⏳ In Progress | Coverage expanding |
|
||||
| Rate limit tests | QA | ⏳ In Progress | |
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|-----------|
|
||||
| **Language** | TypeScript (Node.js ≥20) |
|
||||
| **API** | Fastify 5 (CORS, Helmet, rate-limit, Swagger, multipart) |
|
||||
| **Frontend** | SolidJS + Vite |
|
||||
| **Mobile** | React Native / Expo SDK 51 |
|
||||
| **Extension** | Chrome Manifest V3 |
|
||||
| **Databases** | PostgreSQL 15/16 (Prisma ORM) + Turso/SQLite (Drizzle) |
|
||||
| **Cache / Queue** | Redis + BullMQ |
|
||||
| **Auth** | NextAuth.js + JWT |
|
||||
| **Billing** | Stripe |
|
||||
| **Email** | Resend |
|
||||
| **Push** | Firebase Cloud Messaging + APNs |
|
||||
| **SMS** | Twilio |
|
||||
| **Analytics** | Mixpanel / Segment |
|
||||
| **Monitoring** | Datadog APM + Sentry |
|
||||
| **ML Models** | ECAPA-TDNN (voice), BERT (SMS), FAISS (vector index) |
|
||||
| **Infrastructure** | Terraform (AWS ECS Fargate, RDS, ElastiCache, S3, ALB) |
|
||||
| **CI/CD** | GitHub Actions |
|
||||
| **Monorepo** | pnpm workspaces + Turborepo |
|
||||
| **Testing** | Vitest, Jest, k6 |
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js >= 20.0.0
|
||||
- pnpm >= 9.0.0
|
||||
- Docker & Docker Compose
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Install all dependencies
|
||||
pnpm install
|
||||
|
||||
# Start local infrastructure (Postgres, Redis, Mailhog)
|
||||
docker compose up -d
|
||||
|
||||
# Copy environment variables
|
||||
cp .env.example .env
|
||||
|
||||
# Run database migrations
|
||||
pnpm db:migrate
|
||||
|
||||
# Start all development servers
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
This launches the API server, all microservices, and the web frontend concurrently via Turborepo.
|
||||
|
||||
---
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
# Build all packages and services
|
||||
pnpm build
|
||||
|
||||
# Build individual Docker images
|
||||
docker build -f packages/api/Dockerfile -t shieldai-api .
|
||||
docker build -f services/spamshield/Dockerfile -t shieldai-spamshield .
|
||||
docker build -f services/darkwatch/Dockerfile -t shieldai-darkwatch .
|
||||
docker build -f services/voiceprint/Dockerfile -t shieldai-voiceprint .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pnpm test
|
||||
|
||||
# With coverage
|
||||
pnpm test:coverage
|
||||
|
||||
# Individual service tests
|
||||
pnpm test --filter @shieldai/spamshield
|
||||
pnpm test --filter @shieldai/darkwatch
|
||||
pnpm test --filter @shieldai/voiceprint
|
||||
pnpm test --filter @shieldai/hometitle
|
||||
|
||||
# Integration & E2E
|
||||
cd packages/integration-tests && pnpm test
|
||||
cd packages/integration-tests && pnpm test:e2e
|
||||
|
||||
# Load tests (requires k6)
|
||||
cd scripts/load-test && ./run-all.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Production
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d
|
||||
```
|
||||
|
||||
### CI/CD Pipeline (GitHub Actions)
|
||||
|
||||
| Event | Deploy To |
|
||||
|-------|-----------|
|
||||
| Push to `main` | Staging |
|
||||
| GitHub Release created | Production |
|
||||
|
||||
Pipeline stages: `lint` → `typecheck` → `test` → `Docker build` → `push to GHCR` → `Terraform apply` → `ECS deploy` → `health check` → auto-rollback on failure.
|
||||
|
||||
### Infrastructure
|
||||
|
||||
All infrastructure lives in `infra/` and is managed with Terraform:
|
||||
|
||||
- **Compute**: AWS ECS Fargate (API + services + workers)
|
||||
- **Database**: RDS PostgreSQL 15/16
|
||||
- **Cache**: ElastiCache Redis
|
||||
- **Storage**: S3 (reports, audio samples)
|
||||
- **Networking**: VPC, ALB, security groups
|
||||
- **Observability**: CloudWatch + Datadog
|
||||
- **Secrets**: AWS Secrets Manager
|
||||
|
||||
See `infra/README.md` and `infra/ROLLBACK.md` for detailed operational runbooks.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
shieldai/
|
||||
├── packages/ # Shared libraries (20 packages)
|
||||
│ ├── api/ # Fastify API server
|
||||
│ ├── core/ # Core shared logic
|
||||
│ ├── db/ # Prisma schemas (v6)
|
||||
│ ├── shared-db/ # Prisma schemas (v5)
|
||||
│ ├── shared-auth/ # NextAuth.js
|
||||
│ ├── shared-billing/ # Stripe subscriptions
|
||||
│ ├── shared-notifications/ # Email, Push, SMS
|
||||
│ ├── shared-analytics/ # Mixpanel/Segment
|
||||
│ ├── shared-ui/ # SolidJS components
|
||||
│ ├── shared-utils/ # Utilities
|
||||
│ ├── types/ # Shared TypeScript types
|
||||
│ ├── mobile/ # React Native / Expo app
|
||||
│ ├── extension/ # Browser extension (MV3)
|
||||
│ ├── jobs/ # BullMQ workers
|
||||
│ ├── monitoring/ # Datadog + Sentry
|
||||
│ ├── report/ # PDF generation
|
||||
│ ├── correlation/ # Event correlation
|
||||
│ ├── mobile-api-client/ # RN API client
|
||||
│ └── integration-tests/ # E2E tests
|
||||
├── services/ # Microservices (5)
|
||||
│ ├── voiceprint/ # Voice cloning detection
|
||||
│ ├── darkwatch/ # Dark web monitoring
|
||||
│ ├── spamshield/ # Spam call/SMS blocking
|
||||
│ ├── hometitle/ # Home title fraud
|
||||
│ └── removebrokers/ # Data broker removal
|
||||
├── infra/ # Terraform (AWS)
|
||||
├── docs/ # Documentation
|
||||
├── plans/ # Product & technical plans
|
||||
├── scripts/ # Utility scripts
|
||||
├── load-tests/ # k6 load test scripts
|
||||
├── assets/ # Ad creative assets
|
||||
└── server/ # Legacy WebSocket server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Location |
|
||||
|----------|----------|
|
||||
| Product Plan | `plans/SHIELDAI-product-plan.md` |
|
||||
| Technical Architecture | `plans/SHIELDAI-technical-architecture.md` |
|
||||
| Infrastructure | `infra/README.md` |
|
||||
| Rollback Runbook | `infra/ROLLBACK.md` |
|
||||
| Stripe Integration | `docs/STRIPE_INTEGRATION.md` |
|
||||
| Push Notifications | `docs/PUSH_NOTIFICATIONS_SETUP.md` |
|
||||
| Mobile Push Integration | `docs/MOBILE_PUSH_INTEGRATION.md` |
|
||||
| Mixpanel Analytics | `docs/MIXPANEL_ANALYTICS.md` |
|
||||
| Code Review Workflow | `shieldai-workflow.md` |
|
||||
|
||||
---
|
||||
|
||||
## Subscription Tiers
|
||||
|
||||
| Feature | Basic (Free) | Plus ($9.99/mo) | Premium ($24.99/mo) |
|
||||
|---------|:------------:|:----------------:|:-------------------:|
|
||||
| Dark web scans | Limited | Unlimited | Unlimited |
|
||||
| Spam detection | Basic | AI-powered | AI-powered |
|
||||
| Voice cloning detection | — | Family | Family |
|
||||
| SSN monitoring | — | — | ✅ |
|
||||
| Home title protection | — | — | ✅ |
|
||||
| Real-time blocking | — | — | ✅ |
|
||||
| 24/7 support | — | — | ✅ |
|
||||
@@ -1,83 +0,0 @@
|
||||
# ShieldAI Code Review Workflow
|
||||
|
||||
## Current State (as of May 2, 2026)
|
||||
|
||||
### PR Backlog Status
|
||||
- **Open PRs**: 0 (pending commits pushed to master)
|
||||
- **Pending commits**: 1 commit pushed (FRE-4604) — remaining 6 were previously pushed
|
||||
- **Last review cycle**: FRE-4500, FRE-4499, FRE-4612 (security findings — all done)
|
||||
- **Branch protection**: Configured (see `branch-protection-rules.yaml`)
|
||||
- **PR template**: Configured (`.gitea/pull_request_templates/default.md`)
|
||||
|
||||
### Resolved Bottlenecks
|
||||
1. ✅ PR-based workflow established with PR template
|
||||
2. ✅ Branch protection rules documented and configured
|
||||
3. ✅ Code review checklist integrated into PR template
|
||||
4. ✅ Security review findings integrated (FRE-4499, FRE-4500, FRE-4612 all done)
|
||||
|
||||
## PR Process
|
||||
|
||||
1. **Feature branch creation** from `gt/master`
|
||||
2. **Development commits** with conventional commit format (include issue ID: `FRE-XXXX: description`)
|
||||
3. **PR creation** against `gt/master`
|
||||
4. **Required reviews**:
|
||||
- Code Reviewer — all PRs
|
||||
- Security Reviewer — for security-sensitive changes
|
||||
5. **CI checks** pass (lint, typecheck, test)
|
||||
6. **Merge** via squash or rebase
|
||||
|
||||
### Code Review Checklist
|
||||
|
||||
- [ ] Security impact assessment
|
||||
- [ ] Test coverage verification
|
||||
- [ ] Type checking (TypeScript)
|
||||
- [ ] Linting compliance
|
||||
- [ ] Documentation updates
|
||||
- [ ] Breaking changes documented
|
||||
- [ ] Backward compatibility verified
|
||||
|
||||
### Branch Protection Rules
|
||||
|
||||
See `branch-protection-rules.yaml` for the full configuration. Summary:
|
||||
|
||||
- **Protected branch**: `gt/master`
|
||||
- **Required reviews**: 1 approved review before merge
|
||||
- **Required status checks**: lint, typecheck, test
|
||||
- **Enforce admins**: false (admins can bypass during emergencies)
|
||||
- **Allow force pushes**: true (for recovery scenarios)
|
||||
|
||||
## Review Assignment Policy
|
||||
|
||||
| Change Type | Required Reviewers |
|
||||
|-------------|-------------------|
|
||||
| General code | Code Reviewer |
|
||||
| Security-critical | Code Reviewer + Security Reviewer |
|
||||
| API contracts | Code Reviewer + CTO |
|
||||
| Database schema | Code Reviewer + Senior Engineer |
|
||||
|
||||
## Review Pipeline
|
||||
|
||||
```
|
||||
Engineer implements → marks in_review → Security Reviewer reviews → Code Reviewer reviews → Done
|
||||
```
|
||||
|
||||
## Metrics to Track
|
||||
|
||||
- PR cycle time (creation to merge)
|
||||
- Review turnaround time
|
||||
- PR size (lines changed)
|
||||
- Review comments per PR
|
||||
- Merge conflict frequency
|
||||
|
||||
## Contribution Guidelines
|
||||
|
||||
1. Always create a feature branch from `gt/master`
|
||||
2. Use conventional commit format: `type(scope): description (FRE-XXXX)`
|
||||
3. Include tests for new functionality
|
||||
4. Update documentation for API changes
|
||||
5. Run lint and typecheck before pushing
|
||||
6. Create PR with filled template before requesting review
|
||||
7. Address all review comments before merge
|
||||
|
||||
---
|
||||
*Updated from FRE-4556 audit, implemented in FRE-4661*
|
||||
Reference in New Issue
Block a user