Add rollback procedure documentation and testing scripts (FRE-4808)
- infra/ROLLBACK.md: comprehensive rollback runbook with ECS, Docker Compose, database migration, blue-green, and emergency rollback procedures - infra/scripts/rollback.sh: enhanced ECS rollback with validation, logging, health verification, and per-service rollback support - infra/scripts/rollback-compose.sh: Docker Compose rollback for local/staging - infra/scripts/rollback-migration.sh: Drizzle migration rollback with AWS Secrets Manager integration - infra/scripts/test-rollback.sh: automated test suite (51 tests) - Updated infra/README.md to reference ROLLBACK.md Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
@@ -14,7 +14,9 @@
|
||||
│ ├── staging/main.tf # Staging environment config
|
||||
│ └── production/main.tf # Production environment config
|
||||
└── scripts/
|
||||
└── rollback.sh # Manual rollback script
|
||||
├── rollback.sh # ECS service rollback (AWS)
|
||||
├── rollback-compose.sh # Docker Compose rollback (local/staging)
|
||||
└── rollback-migration.sh # Database migration rollback
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -75,31 +77,28 @@ terraform apply -var-file=terraform.tfvars.example
|
||||
|
||||
## Rollback
|
||||
|
||||
### Automatic (CI/CD)
|
||||
The deploy workflow triggers automatic rollback when health checks fail:
|
||||
```
|
||||
deploy-ecs → health-check (failure) → rollback
|
||||
```
|
||||
See **[ROLLBACK.md](./ROLLBACK.md)** for the complete rollback runbook, including:
|
||||
|
||||
- ECS service rollback (automated + manual)
|
||||
- Docker Compose rollback (local / staging)
|
||||
- Database migration rollback (Drizzle)
|
||||
- Blue-green deployment rollback
|
||||
- RDS point-in-time recovery
|
||||
- Automated rollback triggers and health checks
|
||||
- Emergency rollback runbook
|
||||
- Testing checklist
|
||||
|
||||
### Quick Reference
|
||||
|
||||
### Manual
|
||||
```bash
|
||||
# Rollback specific service
|
||||
cd infra/scripts
|
||||
./rollback.sh staging api
|
||||
# ECS service rollback (AWS)
|
||||
./infra/scripts/rollback.sh <environment> <service|all> [--verify]
|
||||
|
||||
# Rollback all services
|
||||
./rollback.sh staging all
|
||||
```
|
||||
# Docker Compose rollback (local/staging)
|
||||
./infra/scripts/rollback-compose.sh <previous_tag>
|
||||
|
||||
### Database Migration Rollback
|
||||
```bash
|
||||
# Run previous migration
|
||||
DATABASE_URL=$(aws secretsmanager get-secret-value \
|
||||
--secret-id shieldai-staging-db-password \
|
||||
--query 'SecretString' --output json | jq -r '.host')
|
||||
|
||||
npx prisma migrate resolve --applied <migration_name>
|
||||
npx prisma migrate deploy
|
||||
# Database migration rollback
|
||||
./infra/scripts/rollback-migration.sh <environment> [--migration <name>]
|
||||
```
|
||||
|
||||
## GitHub Secrets Required
|
||||
|
||||
Reference in New Issue
Block a user