272 lines
8.3 KiB
Markdown
272 lines
8.3 KiB
Markdown
# Firebase Test Lab Integration
|
||
|
||
Automated testing on real physical Android devices using Firebase Test Lab.
|
||
Ensures the Kordant Android app works correctly across a diverse device matrix
|
||
including Pixel, Samsung, and Xiaomi devices.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
firebase-test-lab/
|
||
├── README.md # This file
|
||
├── test_matrix_config.yaml # Device matrix and test configuration
|
||
├── robo_script.json # Robo crawl script (guided UI navigation)
|
||
├── run_robo_tests.sh # Run Robo exploratory tests
|
||
├── run_instrumentation_tests.sh # Run instrumentation (UI) tests
|
||
├── run_all_tests.sh # Run full test suite
|
||
└── download_results.sh # Download and analyze test results
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
1. **Google Cloud Project** with Firebase enabled
|
||
2. **Blaze plan** (pay-as-you-go) — Test Lab is free for the first 100 device-minutes/day on physical devices
|
||
3. **gcloud CLI** installed and authenticated
|
||
4. **Service account** with `Firebase Test Lab Admin` role
|
||
|
||
### Installation
|
||
|
||
```bash
|
||
# Install gcloud CLI (macOS)
|
||
brew install --cask google-cloud-sdk
|
||
|
||
# Authenticate
|
||
gcloud auth login
|
||
gcloud auth application-default login
|
||
|
||
# Verify
|
||
gcloud firebase test android models list
|
||
```
|
||
|
||
### Firebase Project Setup
|
||
|
||
1. Create a Firebase project at https://console.firebase.google.com
|
||
2. Enable the Blaze (pay-as-you-go) plan
|
||
3. Optionally link to Google Play Console for deep Play Store integration
|
||
4. Create a service account and download JSON key:
|
||
- IAM & Admin → Service Accounts → Create Service Account
|
||
- Role: `Firebase Test Lab Admin` (roles/firebase.testlab.admin)
|
||
- Create and download JSON key
|
||
|
||
## Device Matrix
|
||
|
||
The app is tested on 5 devices across 2 orientations and 2 locales
|
||
(20 device/locale/orientation combinations total):
|
||
|
||
| Device | Model ID | API | Screen | RAM | Target |
|
||
|--------|----------|-----|--------|-----|--------|
|
||
| Pixel 6 | `Pixel6` | 33 | 1080×2400 | 8GB | Primary target |
|
||
| Pixel 4 | `Pixel4` | 30 | 1080×2280 | 6GB | Older device |
|
||
| Galaxy S21 | `GalaxyS21` | 31 | 1080×2400 | 8GB | Samsung |
|
||
| Redmi Note 8 | `RedmiNote8` | 29 | 1080×2340 | 4GB | Xiaomi / budget |
|
||
| Aquest M2 | `AquestM2` | 28 | 720×1280 | 2GB | Low-end / minimum spec |
|
||
|
||
**Orientations:** portrait, landscape
|
||
**Locales:** en_US (English US), es_ES (Spanish Spain)
|
||
|
||
## Running Tests
|
||
|
||
### 1. Build the app
|
||
|
||
```bash
|
||
cd android
|
||
./gradlew :app:assembleProdRelease :app:assembleProdDebugAndroidTest
|
||
```
|
||
|
||
### 2. Run Robo Tests (exploratory crash detection)
|
||
|
||
Robo tests automatically crawl the app UI without requiring any test code.
|
||
They detect crashes, ANRs, and UI rendering issues.
|
||
|
||
```bash
|
||
cd android/firebase-test-lab
|
||
./run_robo_tests.sh --project-id kordant-android
|
||
```
|
||
|
||
Options:
|
||
- `--project-id` — Firebase project ID (default: `kordant-android`)
|
||
- `--app-aab` — Path to AAB (preferred, more accurate)
|
||
- `--app-apk` — Path to APK (fallback)
|
||
- `--robo-script` — Path to robo crawl script
|
||
- `--timeout` — Max crawl time in seconds (default: 600)
|
||
- `--dry-run` — Preview command without executing
|
||
|
||
### 3. Run Instrumentation Tests (UI tests with assertions)
|
||
|
||
Runs the existing UI test suite (AuthFlowTest, DashboardUITest, ServiceUITests, etc.)
|
||
across the full device matrix.
|
||
|
||
```bash
|
||
cd android/firebase-test-lab
|
||
./run_instrumentation_tests.sh --project-id kordant-android
|
||
```
|
||
|
||
Options:
|
||
- `--project-id` — Firebase project ID
|
||
- `--app-apk` — Path to app APK (auto-detected)
|
||
- `--test-apk` — Path to test APK (auto-detected)
|
||
- `--dry-run` — Preview command without executing
|
||
|
||
### 4. Run Full Test Suite
|
||
|
||
```bash
|
||
cd android/firebase-test-lab
|
||
./run_all_tests.sh --project-id kordant-android
|
||
```
|
||
|
||
Options:
|
||
- `--skip-build` — Skip Gradle build step
|
||
- `--skip-robo` — Skip Robo tests
|
||
- `--skip-instr` — Skip instrumentation tests
|
||
- `--dry-run` — Preview commands without executing
|
||
|
||
### 5. Download Results
|
||
|
||
```bash
|
||
cd android/firebase-test-lab
|
||
./download_results.sh --project-id kordant-android --download-all
|
||
```
|
||
|
||
This downloads:
|
||
- Test result XMLs (JUnit format)
|
||
- Screenshots (PNG)
|
||
- Test videos (MP4)
|
||
- Performance metrics (JSON)
|
||
- Logcat output
|
||
- Crawl maps (Robo test UI exploration paths)
|
||
|
||
## CI Integration
|
||
|
||
The GitHub Actions workflow at `.github/workflows/firebase-test-lab.yml`
|
||
runs automatically on pushes and PRs that modify Android code.
|
||
|
||
### CI Pipeline Flow
|
||
|
||
1. **Build job** — Compiles release and test APKs
|
||
2. **Robo Tests job** — Runs crash/ANR detection on all 20 device configurations
|
||
3. **Instrumentation Tests job** — Runs UI test suite on all 20 device configurations
|
||
4. **Test Summary job** — Collects results and determines pass/fail
|
||
|
||
### GitHub Secrets Required
|
||
|
||
| Secret | Description |
|
||
|--------|-------------|
|
||
| `GCP_SA_KEY_TEST_LAB` | Service account JSON key with Test Lab admin role |
|
||
| `FIREBASE_PROJECT_ID` | Firebase project ID (default: `kordant-android`) |
|
||
|
||
### Adding to CI
|
||
|
||
The workflow triggers on:
|
||
- Push to `main` with Android changes
|
||
- PR to `main` with Android changes
|
||
- Manual trigger via `workflow_dispatch`
|
||
|
||
To block release builds on test failures, add the test-summary job as a
|
||
required check in your branch protection rules.
|
||
|
||
## Robo Test Script
|
||
|
||
The `robo_script.json` file guides the Robo crawler through the app's
|
||
critical user flow:
|
||
|
||
1. Wait for splash screen
|
||
2. Click "Get Started" on the onboarding screen
|
||
3. Click "Sign In" on the login screen
|
||
4. Enter email and password
|
||
5. Submit sign-in
|
||
6. Navigate through: Dashboard → Services → Alerts → Settings
|
||
|
||
This ensures the crawler reaches authenticated screens. The test user
|
||
credentials are injected via `${ROBO_ID}` for unique user per test run.
|
||
|
||
## Test Accounts
|
||
|
||
Robo tests support test accounts for automatic login during the crawl.
|
||
Configure credentials securely:
|
||
|
||
```bash
|
||
gcloud firebase test android run \
|
||
--type robo \
|
||
--app app.apk \
|
||
--device model=Pixel6,version=33,locale=en_US,orientation=portrait \
|
||
--test-accounts username=test@kordant.com,password=$ROBO_PASSWORD
|
||
```
|
||
|
||
For CI, store credentials in GitHub Secrets and pass as environment variables.
|
||
|
||
## Analyzing Results
|
||
|
||
### In Firebase Console
|
||
|
||
1. Navigate to https://console.firebase.google.com/project/YOUR_PROJECT/testlab
|
||
2. View test matrices grouped by history name
|
||
3. Click on a matrix to see per-device results
|
||
4. Watch test videos to identify UI issues
|
||
5. Review screenshots for visual regressions
|
||
6. Check performance metrics for responsiveness
|
||
|
||
### Performance Budget
|
||
|
||
| Metric | Target | Device |
|
||
|--------|--------|--------|
|
||
| Cold start | < 1500ms | Pixel 6 |
|
||
| Warm start | < 1000ms | Pixel 6 |
|
||
| Robo crawl | Complete in < 10min per device | All |
|
||
| No crashes | 0 crashes | All |
|
||
| No ANRs | 0 ANRs | All |
|
||
|
||
### Device-Specific Issues to Watch
|
||
|
||
- **Low-end devices (API 28, 2GB RAM):** Check for OOM, slow rendering, lag
|
||
- **Xiaomi:** Check for MIUI-specific permission quirks
|
||
- **Samsung:** Check for One UI theme compatibility
|
||
- **Landscape:** Verify proper layout adaptation
|
||
- **Spanish locale:** Check for text truncation or layout overflow
|
||
|
||
## Troubleshooting
|
||
|
||
### "Permission denied" when running scripts
|
||
|
||
```bash
|
||
chmod +x android/firebase-test-lab/*.sh
|
||
```
|
||
|
||
### "No authenticated account" error
|
||
|
||
```bash
|
||
gcloud auth login
|
||
gcloud auth application-default login
|
||
```
|
||
|
||
### "Project not found" error
|
||
|
||
Verify the project exists and has the Blaze plan enabled:
|
||
```bash
|
||
gcloud projects list
|
||
gcloud firebase test android models list --project YOUR_PROJECT_ID
|
||
```
|
||
|
||
### "Quota exceeded" error
|
||
|
||
Firebase Test Lab has daily quotas. Check usage in the Firebase Console.
|
||
The free tier provides 100 device-minutes/day on physical devices.
|
||
|
||
### Test APK not found
|
||
|
||
Build the test APK first:
|
||
```bash
|
||
cd android
|
||
./gradlew :app:assembleProdDebugAndroidTest
|
||
```
|
||
|
||
## Best Practices
|
||
|
||
1. **Run Robo tests first** — They're free-form and catch crashes without test code
|
||
2. **Always test on low-end devices** — Many issues only appear on 2GB RAM devices
|
||
3. **Review screenshots** — Visual issues are common across device families
|
||
4. **Watch videos of failures** — The video shows exactly what led to the crash
|
||
5. **Run on release builds** — Debug builds may mask issues
|
||
6. **Use AAB for Robo tests** — More accurate representation of Play Store installs
|
||
7. **Set --fail-fast for CI** — Stop on first failure to save device-minutes
|
||
8. **Archive results** — Keep screenshots and videos for regression comparison
|