Add rollback procedure documentation and testing scripts (FRE-4808)

- infra/ROLLBACK.md: comprehensive rollback runbook with ECS, Docker Compose,
  database migration, blue-green, and emergency rollback procedures
- infra/scripts/rollback.sh: enhanced ECS rollback with validation, logging,
  health verification, and per-service rollback support
- infra/scripts/rollback-compose.sh: Docker Compose rollback for local/staging
- infra/scripts/rollback-migration.sh: Drizzle migration rollback with
  AWS Secrets Manager integration
- infra/scripts/test-rollback.sh: automated test suite (51 tests)
- Updated infra/README.md to reference ROLLBACK.md

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
2026-05-09 06:27:31 -04:00
parent 540ca5ebad
commit bce4787802
6 changed files with 1394 additions and 40 deletions

View File

@@ -14,7 +14,9 @@
│ ├── staging/main.tf # Staging environment config
│ └── production/main.tf # Production environment config
└── scripts/
── rollback.sh # Manual rollback script
── rollback.sh # ECS service rollback (AWS)
├── rollback-compose.sh # Docker Compose rollback (local/staging)
└── rollback-migration.sh # Database migration rollback
## Quick Start
@@ -75,31 +77,28 @@ terraform apply -var-file=terraform.tfvars.example
## Rollback
### Automatic (CI/CD)
The deploy workflow triggers automatic rollback when health checks fail:
```
deploy-ecs → health-check (failure) → rollback
```
See **[ROLLBACK.md](./ROLLBACK.md)** for the complete rollback runbook, including:
- ECS service rollback (automated + manual)
- Docker Compose rollback (local / staging)
- Database migration rollback (Drizzle)
- Blue-green deployment rollback
- RDS point-in-time recovery
- Automated rollback triggers and health checks
- Emergency rollback runbook
- Testing checklist
### Quick Reference
### Manual
```bash
# Rollback specific service
cd infra/scripts
./rollback.sh staging api
# ECS service rollback (AWS)
./infra/scripts/rollback.sh <environment> <service|all> [--verify]
# Rollback all services
./rollback.sh staging all
```
# Docker Compose rollback (local/staging)
./infra/scripts/rollback-compose.sh <previous_tag>
### Database Migration Rollback
```bash
# Run previous migration
DATABASE_URL=$(aws secretsmanager get-secret-value \
--secret-id shieldai-staging-db-password \
--query 'SecretString' --output json | jq -r '.host')
npx prisma migrate resolve --applied <migration_name>
npx prisma migrate deploy
# Database migration rollback
./infra/scripts/rollback-migration.sh <environment> [--migration <name>]
```
## GitHub Secrets Required