# 32. Migration Safety & Rollback Procedures

meta:
  id: web-production-32
  feature: web-production
  priority: P1
  depends_on: []
  tags: [database, reliability, production]

objective:
- Ensure database migrations are safe, reversible, and won't cause downtime or data loss in production

deliverables:
- Migration safety guidelines
- Backward-compatible migration policy
- Rollback scripts for each migration
- Migration testing in staging

steps:
1. Create migration safety guidelines:
   - Document in docs/MIGRATIONS.md
   - Additive changes only in production (add columns, create tables)
   - No destructive changes during deployment (no DROP COLUMN)
   - Two-phase migrations for destructive changes:
     - Phase 1: Add new column/table, deploy code to use it
     - Phase 2: Remove old column/table after code stable
2. Audit existing migrations:
   - Review all drizzle migrations in web/src/server/db/
   - Check for any destructive operations
   - Add rollback scripts where missing
3. Implement migration testing:
   - Run migrations against staging database copy
   - Verify app works after migration
   - Test rollback script
   - Measure migration duration (must be <30 seconds)
4. Add migration safety checks:
   - CI check: verify no destructive migrations in PR
   - Pre-deploy: dry-run migration in production
   - Post-deploy: verify migration applied successfully
5. Document rollback procedures:
   - Step-by-step rollback for each migration
   - Database backup before migration
   - Code rollback procedure
   - Data recovery steps if needed
6. Add migration monitoring:
   - Log migration start, duration, success/failure
   - Alert on migration failure
   - Track migration duration trends
7. Set up migration automation:
   - GitHub Action to run migrations on staging deploy
   - Manual approval for production migrations
   - Automated rollback on migration failure

tests:
- Unit: Test migration scripts in isolation
- Integration: Test migration on staging database
- Rollback: Test rollback procedure

acceptance_criteria:
- All production migrations are additive-only
- Two-phase migration process documented for destructive changes
- Rollback script exists for every migration
- Migrations tested on staging before production
- Migration duration <30 seconds
- Automated CI check preventing destructive migrations
- Backup taken before every production migration
- Migration failure triggers automatic alert and rollback

validation:
- Review migration history → no destructive changes in production
- Test rollback → database restored to previous state
- Run destructive migration in PR → CI blocks merge
- Check migration logs → all migrations completed successfully

notes:
- Drizzle migrations are generally safe but review generated SQL
- Use drizzle-kit generate with --custom for complex migrations
- Consider using gh-ost or pt-online-schema-change for large tables
- Always have a database backup before running production migrations