Mike/Kordant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
Michael Freno
2026-05-18 09:00:23 -04:00
parent f118d3a4f3
commit 56c4b1bc03
2 changed files with 309 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

309
README.md Normal file
View File

@@ -0,0 +1,309 @@
# ShieldAI
**Multi-layered consumer identity protection against predatory AI-driven scams.**
ShieldAI combines three detection engines — voice cloning detection, dark web monitoring, and real-time spam classification — to give consumers proactive defense against modern identity fraud.
---
## The Pitch
Scammers are weaponizing AI at scale: voice clones that sound exactly like your family, hyper-personalized phishing messages that bypass filters, and synthetic identities that exploit stolen data within hours of a breach. Legacy credit monitoring is reactive — it tells you after the damage is done.
ShieldAI flips the model. We detect the scam _as it happens_:
- **VoicePrint** analyzes inbound calls in real time to flag synthetic AI-generated voices before you're socially engineered.
- **DarkWatch** continuously monitors dark web forums, breach databases, and data broker caches — notifying you the moment your credentials, phone, or SSN surface.
- **SpamShield** intercepts and classifies spam calls and SMS at the network level, blocking threats before they reach your phone.
Backed by ML models (ECAPA-TDNN, BERT) and a real-time alert pipeline, ShieldAI gives consumers enterprise-grade threat detection for their personal life.
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────┐
│ Clients │
│ Mobile (Expo/RN) │ Web (SolidJS) │ Browser Extension │
└──────────┬──────────────────────────────┬───────────────┘
│ REST + WebSocket │
▼ ▼
┌──────────────────────┐ ┌──────────────────────────────┐
│ API Gateway │ │ WebSocket Alert Server │
│ (Fastify 5) │ │ (Real-time push) │
└──────┬───────┬───────┘ └──────────────────────────────┘
│ │
▼ ▼
┌──────────┐ ┌─────────────────────────────────────────────┐
│ Auth │ │ Microservices │
│(NextAuth)│ │ VoicePrint │ DarkWatch │ SpamShield │
└──────────┘ │ HomeTitle │ RemoveBrokers │
└──────────┬──────────┬───────────────────────┘
│ │
┌────────▼──────────▼────────┐
│ Background Workers │
│ (BullMQ + Redis) │
└────────┬───────────────────┘
│
┌────────▼────────┐
│ PostgreSQL │
│ + Redis │
└─────────────────┘
```
---
## Features & Implementation Status
| Feature | Service |Status | Notes |
|---------|---------|--------|-------|
| Voice enrollment & profile management | VoicePrint | ✅ Done | Register family voice profiles |
| Audio preprocessing (VAD, noise reduction) | VoicePrint | ✅ Done | WebRTC VAD + RNNoise |
| Synthetic voice detection (ECAPA-TDNN) | VoicePrint | ✅ Done | FAISS vector index for matching |
| Real-time streaming audio analysis | VoicePrint | ✅ Done | WebSocket-based |
| Batch audio analysis | VoicePrint | ✅ Done | Configurable confidence thresholds per tier |
| HIBP breach checking | DarkWatch | ✅ Done | Email + password breach lookup |
| Dark web multi-source scanning | DarkWatch | ✅ Done | HIBP, SecurityTrails, Censys, Shodan, forums |
| Watch list management | DarkWatch | ✅ Done | Emails, phones, SSN (hashed) |
| Scheduled + real-time scanning | DarkWatch | ✅ Done | Tier-based frequency |
| Fuzzy matching engine | DarkWatch | ✅ Done | Levenshtein + exact matching |
| Severity-scored alert pipeline | DarkWatch | ✅ Done | Dedup pipeline |
| PDF report generation | DarkWatch | ✅ Done | Handlebars + PDFKit |
| Number reputation (Hiya/Truecaller) | SpamShield | ✅ Done | Circuit breaker pattern |
| SMS classification (BERT) | SpamShield | ✅ Done | ML-based spam detection |
| Call analysis rule engine | SpamShield | ✅ Done | Multi-layer scoring |
| Real-time carrier interception | SpamShield | ⏳ In Progress | Twilio/Plivo integration |
| Real-time WebSocket alerts | SpamShield | ✅ Done | Alert broadcasting |
| User feedback loop (FP/FN) | SpamShield | ✅ Done | Metadata validation |
| Phone validation (E.164) | SpamShield | ✅ Done | Normalization |
| Audit logging | SpamShield | ✅ Done | All decisions logged |
| SpamShield rate limiting | SpamShield | ⏳ In Progress | Per-endpoint + global |
| SpamShield route optimization | SpamShield | ⏳ In Progress | Route consolidation |
| Feature flags | All | ✅ Done | Env-variable toggles |
| Property record matching | HomeTitle | ✅ Done | Fuzzy string matching |
| Change detection (ownership, liens) | HomeTitle | ✅ Done | County deed scanning |
| Watchlist management | HomeTitle | ✅ Done | |
| Scheduled county deed scanning | HomeTitle | ✅ Done | |
| Alert pipeline | HomeTitle | ✅ Done | Severity classification |
| Data broker removal requests | RemoveBrokers | ✅ Done | |
| Broker API integration | RemoveBrokers | ✅ Done | With caching |
| User auth (JWT, RBAC) | Shared | ✅ Done | NextAuth.js |
| Family group management | Shared | ✅ Done | |
| Stripe subscriptions & billing | Shared | ✅ Done | Tier-based feature gating |
| Email (Resend) | Shared | ✅ Done | Transactional + marketing |
| Push notifications (FCM/APNs) | Shared | ✅ Done | Android + iOS |
| SMS (Twilio) | Shared | ✅ Done | |
| Mixpanel analytics (30+ events) | Shared | ✅ Done | KPI tracking |
| Datadog APM + Sentry | Shared | ✅ Done | Full observability |
| Cross-service event correlation | Shared | ✅ Done | Alert correlation engine |
| Browser extension (MV3) | Extension | ✅ Done | Phishing detection |
| Mobile app (Expo RN) | Mobile | ✅ Done | iOS + Android |
| Shared UI component library | Shared UI | ✅ Done | SolidJS |
| CI/CD pipelines | DevOps | ✅ Done | GitHub Actions |
| Terraform infrastructure | DevOps | ✅ Done | AWS ECS, RDS, ElastiCache |
| Load testing (k6) | DevOps | ✅ Done | VoicePrint + DarkWatch |
| Docker + Compose | DevOps | ✅ Done | Dev + prod configs |
| Integration tests | QA | ⏳ In Progress | Coverage expanding |
| Rate limit tests | QA | ⏳ In Progress | |
---
## Tech Stack
| Layer | Technology |
|-------|-----------|
| **Language** | TypeScript (Node.js ≥20) |
| **API** | Fastify 5 (CORS, Helmet, rate-limit, Swagger, multipart) |
| **Frontend** | SolidJS + Vite |
| **Mobile** | React Native / Expo SDK 51 |
| **Extension** | Chrome Manifest V3 |
| **Databases** | PostgreSQL 15/16 (Prisma ORM) + Turso/SQLite (Drizzle) |
| **Cache / Queue** | Redis + BullMQ |
| **Auth** | NextAuth.js + JWT |
| **Billing** | Stripe |
| **Email** | Resend |
| **Push** | Firebase Cloud Messaging + APNs |
| **SMS** | Twilio |
| **Analytics** | Mixpanel / Segment |
| **Monitoring** | Datadog APM + Sentry |
| **ML Models** | ECAPA-TDNN (voice), BERT (SMS), FAISS (vector index) |
| **Infrastructure** | Terraform (AWS ECS Fargate, RDS, ElastiCache, S3, ALB) |
| **CI/CD** | GitHub Actions |
| **Monorepo** | pnpm workspaces + Turborepo |
| **Testing** | Vitest, Jest, k6 |
---
## Getting Started
### Prerequisites
- Node.js >= 20.0.0
- pnpm >= 9.0.0
- Docker & Docker Compose
### Setup
```bash
# Install all dependencies
pnpm install
# Start local infrastructure (Postgres, Redis, Mailhog)
docker compose up -d
# Copy environment variables
cp .env.example .env
# Run database migrations
pnpm db:migrate
# Start all development servers
pnpm dev
```
This launches the API server, all microservices, and the web frontend concurrently via Turborepo.
---
## Building
```bash
# Build all packages and services
pnpm build
# Build individual Docker images
docker build -f packages/api/Dockerfile -t shieldai-api .
docker build -f services/spamshield/Dockerfile -t shieldai-spamshield .
docker build -f services/darkwatch/Dockerfile -t shieldai-darkwatch .
docker build -f services/voiceprint/Dockerfile -t shieldai-voiceprint .
```
---
## Testing
```bash
# Run all tests
pnpm test
# With coverage
pnpm test:coverage
# Individual service tests
pnpm test --filter @shieldai/spamshield
pnpm test --filter @shieldai/darkwatch
pnpm test --filter @shieldai/voiceprint
pnpm test --filter @shieldai/hometitle
# Integration & E2E
cd packages/integration-tests && pnpm test
cd packages/integration-tests && pnpm test:e2e
# Load tests (requires k6)
cd scripts/load-test && ./run-all.sh
```
---
## Production
### Docker Compose
```bash
docker compose -f docker-compose.prod.yml up -d
```
### CI/CD Pipeline (GitHub Actions)
| Event | Deploy To |
|-------|-----------|
| Push to `main` | Staging |
| GitHub Release created | Production |
Pipeline stages: `lint` → `typecheck` → `test` → `Docker build` → `push to GHCR` → `Terraform apply` → `ECS deploy` → `health check` → auto-rollback on failure.
### Infrastructure
All infrastructure lives in `infra/` and is managed with Terraform:
- **Compute**: AWS ECS Fargate (API + services + workers)
- **Database**: RDS PostgreSQL 15/16
- **Cache**: ElastiCache Redis
- **Storage**: S3 (reports, audio samples)
- **Networking**: VPC, ALB, security groups
- **Observability**: CloudWatch + Datadog
- **Secrets**: AWS Secrets Manager
See `infra/README.md` and `infra/ROLLBACK.md` for detailed operational runbooks.
---
## Project Structure
```
shieldai/
├── packages/ # Shared libraries (20 packages)
│ ├── api/ # Fastify API server
│ ├── core/ # Core shared logic
│ ├── db/ # Prisma schemas (v6)
│ ├── shared-db/ # Prisma schemas (v5)
│ ├── shared-auth/ # NextAuth.js
│ ├── shared-billing/ # Stripe subscriptions
│ ├── shared-notifications/ # Email, Push, SMS
│ ├── shared-analytics/ # Mixpanel/Segment
│ ├── shared-ui/ # SolidJS components
│ ├── shared-utils/ # Utilities
│ ├── types/ # Shared TypeScript types
│ ├── mobile/ # React Native / Expo app
│ ├── extension/ # Browser extension (MV3)
│ ├── jobs/ # BullMQ workers
│ ├── monitoring/ # Datadog + Sentry
│ ├── report/ # PDF generation
│ ├── correlation/ # Event correlation
│ ├── mobile-api-client/ # RN API client
│ └── integration-tests/ # E2E tests
├── services/ # Microservices (5)
│ ├── voiceprint/ # Voice cloning detection
│ ├── darkwatch/ # Dark web monitoring
│ ├── spamshield/ # Spam call/SMS blocking
│ ├── hometitle/ # Home title fraud
│ └── removebrokers/ # Data broker removal
├── infra/ # Terraform (AWS)
├── docs/ # Documentation
├── plans/ # Product & technical plans
├── scripts/ # Utility scripts
├── load-tests/ # k6 load test scripts
├── assets/ # Ad creative assets
└── server/ # Legacy WebSocket server
```
---
## Documentation
| Document | Location |
|----------|----------|
| Product Plan | `plans/SHIELDAI-product-plan.md` |
| Technical Architecture | `plans/SHIELDAI-technical-architecture.md` |
| Infrastructure | `infra/README.md` |
| Rollback Runbook | `infra/ROLLBACK.md` |
| Stripe Integration | `docs/STRIPE_INTEGRATION.md` |
| Push Notifications | `docs/PUSH_NOTIFICATIONS_SETUP.md` |
| Mobile Push Integration | `docs/MOBILE_PUSH_INTEGRATION.md` |
| Mixpanel Analytics | `docs/MIXPANEL_ANALYTICS.md` |
| Code Review Workflow | `shieldai-workflow.md` |
---
## Subscription Tiers
| Feature | Basic (Free) | Plus ($9.99/mo) | Premium ($24.99/mo) |
|---------|:------------:|:----------------:|:-------------------:|
| Dark web scans | Limited | Unlimited | Unlimited |
| Spam detection | Basic | AI-powered | AI-powered |
| Voice cloning detection | — | Family | Family |
| SSN monitoring | — | — | ✅ |
| Home title protection | — | — | ✅ |
| Real-time blocking | — | — | ✅ |
| 24/7 support | — | — | ✅ |

83
shieldai-workflow.md
View File

@@ -1,83 +0,0 @@
# ShieldAI Code Review Workflow
## Current State (as of May 2, 2026)
### PR Backlog Status
- **Open PRs**: 0 (pending commits pushed to master)
- **Pending commits**: 1 commit pushed (FRE-4604) — remaining 6 were previously pushed
- **Last review cycle**: FRE-4500, FRE-4499, FRE-4612 (security findings — all done)
- **Branch protection**: Configured (see `branch-protection-rules.yaml`)
- **PR template**: Configured (`.gitea/pull_request_templates/default.md`)
### Resolved Bottlenecks
1. ✅ PR-based workflow established with PR template
2. ✅ Branch protection rules documented and configured
3. ✅ Code review checklist integrated into PR template
4. ✅ Security review findings integrated (FRE-4499, FRE-4500, FRE-4612 all done)
## PR Process
1. **Feature branch creation** from `gt/master`
2. **Development commits** with conventional commit format (include issue ID: `FRE-XXXX: description`)
3. **PR creation** against `gt/master`
4. **Required reviews**:
- Code Reviewer — all PRs
- Security Reviewer — for security-sensitive changes
5. **CI checks** pass (lint, typecheck, test)
6. **Merge** via squash or rebase
### Code Review Checklist
- [ ] Security impact assessment
- [ ] Test coverage verification
- [ ] Type checking (TypeScript)
- [ ] Linting compliance
- [ ] Documentation updates
- [ ] Breaking changes documented
- [ ] Backward compatibility verified
### Branch Protection Rules
See `branch-protection-rules.yaml` for the full configuration. Summary:
- **Protected branch**: `gt/master`
- **Required reviews**: 1 approved review before merge
- **Required status checks**: lint, typecheck, test
- **Enforce admins**: false (admins can bypass during emergencies)
- **Allow force pushes**: true (for recovery scenarios)
## Review Assignment Policy
| Change Type | Required Reviewers |
|-------------|-------------------|
| General code | Code Reviewer |
| Security-critical | Code Reviewer + Security Reviewer |
| API contracts | Code Reviewer + CTO |
| Database schema | Code Reviewer + Senior Engineer |
## Review Pipeline
```
Engineer implements → marks in_review → Security Reviewer reviews → Code Reviewer reviews → Done
```
## Metrics to Track
- PR cycle time (creation to merge)
- Review turnaround time
- PR size (lines changed)
- Review comments per PR
- Merge conflict frequency
## Contribution Guidelines
1. Always create a feature branch from `gt/master`
2. Use conventional commit format: `type(scope): description (FRE-XXXX)`
3. Include tests for new functionality
4. Update documentation for API changes
5. Run lint and typecheck before pushing
6. Create PR with filled template before requesting review
7. Address all review comments before merge
---
*Updated from FRE-4556 audit, implemented in FRE-4661*