P0: Fix validation bypass - validated properties now override raw properties
P1: Add unit tests for shared-analytics package (3 test files)
P1: Refactor spamshield to use shared-analytics, deprecate duplicate
P2: Normalize phone numbers to E.164 before hashing
P2: Add graceful error handling for missing env vars in config
P3: Add singleton pattern to MixpanelService
P3: Include timestamp in validated properties schema
P0 fixes:
- Add CANCELLED status to RemovalStatus enum (types + Prisma schema)
- Use CANCELLED instead of REJECTED for user-initiated cancellations
- Add null guard for req.broker?.name in GET /request/:id
- Remove unsafe 'as any' casts in RemoveBrokersService.ts
- Add type-safe toPersonalInfo() validator for JSON deserialization
- Type RemovalRequestWithBroker properly in getRemovalStatus()
- Fix alert: any to NormalizedAlertInput in BrokerAlertPipeline
P1 fixes:
- Fix admin role check: remove non-existent 'admin', only check 'support'
- Fix BrokerDefinition.category type from string to BrokerCategory
- Add complete OpenAPI spec for all removebrokers routes and schemas
- Fix Prisma enum casing: snake_case -> UPPERCASE to match TypeScript types
- Add admin auth guard on POST /process endpoint (P0 security)
- Fix DELETE /request/:id to return valid enum status (REJECTED not cancelled)
- Fix brokerName bug: was set to brokerId, now resolves actual broker name
- Add missing BrokerCategory enum export to types package
- Add HOME_TITLE to AlertSource enum
- Replace unsafe 'as any' casts with proper enum imports
- Fix broker ID with space (familytree Now -> familytreenow)
- Add missing Prisma relation fields for RemovalRequest and BrokerListing
- Add FALSE_POSITIVE to CorrelationStatus enum
Co-Authored-By: Paperclip <noreply@paperclip.ing>
New service for helping clients remove personal listings from data broker sites.
Service features:
- BrokerRegistry: Catalog of 20+ data brokers with removal methods
- RemoveBrokersService: Core service for scanning, creating removal requests,
submitting removals, and verifying completions
- RemoveBrokersScheduler: Automated processing of pending removals and
verification of completed removals
- BrokerAlertPipeline: Alert integration for listing discoveries and removal status
API endpoints (/removebrokers):
- GET /brokers - List available data brokers
- GET /status - Get removal request status and stats
- POST /scan - Scan for personal listings across brokers
- POST /request - Create a new removal request
- GET /request/:id - Get specific removal request details
- DELETE /request/:id - Cancel a removal request
- POST /process - Trigger processing of pending removals
- POST /verify/:id - Manually verify a removal completion
DB models: InfoBroker, RemovalRequest, BrokerListing
Types: BrokerStatus, RemovalStatus, RemovalMethod, and related interfaces
- ATTOM Property API integration for structured property data
- USPS address standardization via API
- County clerk/recorder feed scraping for deed changes and liens
- Rate limiting, caching, and retry logic
- Unit tests for each data source adapter
- PropertyRecord, CountyDeedRecord, DataSourceType types in types.ts
- Consolidated type exports in index.ts
Co-Authored-By: Paperclip <noreply@paperclip.ing>
P3-1: Replace envSchema.parse() with safeParse() + default fallback to
avoid module-level crash when env vars are missing.
P3-3: Add fs.existsSync check on ECAPA_TDNN_MODEL_PATH at startup
with warning log when model path is missing.
P3-4: Add Zod strict() mode to env schema to catch typos in env
var names (extra keys now produce validation errors).
P1-6: Confirmed resolved - voiceprint.service.ts already imports
VoiceEnrollment/VoiceAnalysis from @shieldai/db (consolidated package).
- S01 (High): Pre-compile regex patterns in RuleEngine.loadActiveRules() and
cache them; eliminate per-evaluation RegExp construction in rule-engine.ts
and spamshield.service.ts (ReDoS mitigation)
- S02 (High): SMS classifier now accepts optional senderPhoneNumber via
SmsClassificationContext; reputation check uses actual sender instead of
hardcoded 'placeholder'
- S03 (Medium): AlertServer (services/spamshield) now enforces JWT auth,
origin allowlist, and max client limit on WebSocket connections
- S04 (Medium): hashPhoneNumber() uses SHA-256 (crypto.createHash) instead
of reversible hex encoding (Buffer.toString('hex'))
- S05 (Medium): DecisionEngine.evaluate() wraps evaluation in Promise.race
with configurable evaluationTimeout; returns fallback decision on timeout
- S06 (Medium): CarrierFactory.getAllCarriers() is now async and properly
awaits isHealthy() promises instead of returning raw Promise objects
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Security findings from April 30 review were claimed fixed but never committed.
Applied all remediations:
HIGH:
- WebhookHandler: fail fast when DARKWATCH_WEBHOOK_SECRET missing instead of defaulting to hardcoded secret
- field-encryption.service: require PII_ENCRYPTION_KEY at startup instead of defaulting
MEDIUM:
- WebhookHandler: make signature required (was optional, accepted unsigned events)
- WebhookHandler: reject unknown event types instead of silently defaulting to SCAN_TRIGGER
- scheduler.routes + webhook.routes: add ownership checks on /:userId endpoints (IDOR)
LOW:
- webhook.routes: generic error responses, full error logged server-side
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Create MixpanelService that uses FieldEncryptionService.hashPhoneNumber()
to SHA-256 hash phone numbers before sending to Mixpanel analytics.
- Implement spamBlocked() method with phone number hashing
- Add 16 unit tests verifying hash correctness and API behavior
- Export service from package index
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Create regex-validation utility with ReDoS detection (nested quantifiers,
overlapping alternations, complexity limits)
- Add @db.VarChar(500) constraint on pattern field in Prisma schema
- Integrate validation in rule-engine at load time and evaluation time
- Add 46 unit tests covering syntax, ReDoS patterns, complexity, edge cases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add runtime flag evaluation from FLAG_<KEY> environment variables
- Add enableCallAnalysis flag check to analyzeCall() and interceptCall()
- Add enableFeedbackLoop flag check to recordFeedback()
- Add 19 tests for feature flag behavior (checkFeatureFlag, getters, service integration)
- Add vitest config and test script to spamshield package
- Created decision-engine.constants.ts with all scoring weights, thresholds, and behavioral scores
- Updated decision-engine.ts to import and use constants instead of inline values
- All 12 hardcoded values now have named, documented constants
- Pre-existing type errors are unrelated to this change
- Implement CircuitBreaker class with CLOSED/OPEN/HALF_OPEN states
- Configurable failure threshold, success threshold, and timeout
- Fallback behavior when circuit opens (returns neutral 0.5 score)
- State change callbacks for monitoring and logging
- Comprehensive metrics tracking (executions, failures, successes, timestamps)
- Update SpamShieldService to use circuit breakers for both Hiya and Truecaller
- Add parallel API calls with graceful degradation
- Export circuit breaker types and service interfaces
- 32 unit tests covering circuit transitions, fallback, and service integration
Co-Authored-By: Paperclip <noreply@paperclip.ing>
