Kordant/android/firebase-test-lab

Firebase Test Lab Integration

Automated testing on real physical Android devices using Firebase Test Lab. Ensures the Kordant Android app works correctly across a diverse device matrix including Pixel, Samsung, and Xiaomi devices.

Architecture

firebase-test-lab/
├── README.md                         # This file
├── test_matrix_config.yaml           # Device matrix and test configuration
├── robo_script.json                  # Robo crawl script (guided UI navigation)
├── run_robo_tests.sh                 # Run Robo exploratory tests
├── run_instrumentation_tests.sh      # Run instrumentation (UI) tests
├── run_all_tests.sh                  # Run full test suite
└── download_results.sh               # Download and analyze test results

Prerequisites

1. Google Cloud Project with Firebase enabled
2. Blaze plan (pay-as-you-go) — Test Lab is free for the first 100 device-minutes/day on physical devices
3. gcloud CLI installed and authenticated
4. Service account with Firebase Test Lab Admin role

Installation

# Install gcloud CLI (macOS)
brew install --cask google-cloud-sdk

# Authenticate
gcloud auth login
gcloud auth application-default login

# Verify
gcloud firebase test android models list

Firebase Project Setup

1. Create a Firebase project at https://console.firebase.google.com
2. Enable the Blaze (pay-as-you-go) plan
3. Optionally link to Google Play Console for deep Play Store integration
4. Create a service account and download JSON key:
   • IAM & Admin → Service Accounts → Create Service Account
   • Role: Firebase Test Lab Admin (roles/firebase.testlab.admin)
   • Create and download JSON key

Device Matrix

The app is tested on 5 devices across 2 orientations and 2 locales (20 device/locale/orientation combinations total):

Device	Model ID	API	Screen	RAM	Target
Pixel 6	Pixel6	33	1080×2400	8GB	Primary target
Pixel 4	Pixel4	30	1080×2280	6GB	Older device
Galaxy S21	GalaxyS21	31	1080×2400	8GB	Samsung
Redmi Note 8	RedmiNote8	29	1080×2340	4GB	Xiaomi / budget
Aquest M2	AquestM2	28	720×1280	2GB	Low-end / minimum spec

Orientations: portrait, landscape Locales: en_US (English US), es_ES (Spanish Spain)

Running Tests

1. Build the app

cd android
./gradlew :app:assembleProdRelease :app:assembleProdDebugAndroidTest

2. Run Robo Tests (exploratory crash detection)

Robo tests automatically crawl the app UI without requiring any test code. They detect crashes, ANRs, and UI rendering issues.

cd android/firebase-test-lab
./run_robo_tests.sh --project-id kordant-android

Options:

• --project-id — Firebase project ID (default: kordant-android)
• --app-aab — Path to AAB (preferred, more accurate)
• --app-apk — Path to APK (fallback)
• --robo-script — Path to robo crawl script
• --timeout — Max crawl time in seconds (default: 600)
• --dry-run — Preview command without executing

3. Run Instrumentation Tests (UI tests with assertions)

Runs the existing UI test suite (AuthFlowTest, DashboardUITest, ServiceUITests, etc.) across the full device matrix.

cd android/firebase-test-lab
./run_instrumentation_tests.sh --project-id kordant-android

Options:

• --project-id — Firebase project ID
• --app-apk — Path to app APK (auto-detected)
• --test-apk — Path to test APK (auto-detected)
• --dry-run — Preview command without executing

4. Run Full Test Suite

cd android/firebase-test-lab
./run_all_tests.sh --project-id kordant-android

Options:

• --skip-build — Skip Gradle build step
• --skip-robo — Skip Robo tests
• --skip-instr — Skip instrumentation tests
• --dry-run — Preview commands without executing

5. Download Results

cd android/firebase-test-lab
./download_results.sh --project-id kordant-android --download-all

This downloads:

• Test result XMLs (JUnit format)
• Screenshots (PNG)
• Test videos (MP4)
• Performance metrics (JSON)
• Logcat output
• Crawl maps (Robo test UI exploration paths)

CI Integration

The GitHub Actions workflow at .github/workflows/firebase-test-lab.yml runs automatically on pushes and PRs that modify Android code.

CI Pipeline Flow

1. Build job — Compiles release and test APKs
2. Robo Tests job — Runs crash/ANR detection on all 20 device configurations
3. Instrumentation Tests job — Runs UI test suite on all 20 device configurations
4. Test Summary job — Collects results and determines pass/fail

GitHub Secrets Required

Secret	Description
GCP_SA_KEY_TEST_LAB	Service account JSON key with Test Lab admin role
FIREBASE_PROJECT_ID	Firebase project ID (default: kordant-android)

Adding to CI

The workflow triggers on:

• Push to main with Android changes
• PR to main with Android changes
• Manual trigger via workflow_dispatch

To block release builds on test failures, add the test-summary job as a required check in your branch protection rules.

Robo Test Script

The robo_script.json file guides the Robo crawler through the app's critical user flow:

1. Wait for splash screen
2. Click "Get Started" on the onboarding screen
3. Click "Sign In" on the login screen
4. Enter email and password
5. Submit sign-in
6. Navigate through: Dashboard → Services → Alerts → Settings

This ensures the crawler reaches authenticated screens. The test user credentials are injected via ${ROBO_ID} for unique user per test run.

Test Accounts

Robo tests support test accounts for automatic login during the crawl. Configure credentials securely:

gcloud firebase test android run \
  --type robo \
  --app app.apk \
  --device model=Pixel6,version=33,locale=en_US,orientation=portrait \
  --test-accounts username=test@kordant.com,password=$ROBO_PASSWORD

For CI, store credentials in GitHub Secrets and pass as environment variables.

Analyzing Results

In Firebase Console

1. Navigate to https://console.firebase.google.com/project/YOUR_PROJECT/testlab
2. View test matrices grouped by history name
3. Click on a matrix to see per-device results
4. Watch test videos to identify UI issues
5. Review screenshots for visual regressions
6. Check performance metrics for responsiveness

Performance Budget

Metric	Target	Device
Cold start	< 1500ms	Pixel 6
Warm start	< 1000ms	Pixel 6
Robo crawl	Complete in < 10min per device	All
No crashes	0 crashes	All
No ANRs	0 ANRs	All

Device-Specific Issues to Watch

• Low-end devices (API 28, 2GB RAM): Check for OOM, slow rendering, lag
• Xiaomi: Check for MIUI-specific permission quirks
• Samsung: Check for One UI theme compatibility
• Landscape: Verify proper layout adaptation
• Spanish locale: Check for text truncation or layout overflow

Troubleshooting

"Permission denied" when running scripts

chmod +x android/firebase-test-lab/*.sh

"No authenticated account" error

gcloud auth login
gcloud auth application-default login

"Project not found" error

Verify the project exists and has the Blaze plan enabled:

gcloud projects list
gcloud firebase test android models list --project YOUR_PROJECT_ID

"Quota exceeded" error

Firebase Test Lab has daily quotas. Check usage in the Firebase Console. The free tier provides 100 device-minutes/day on physical devices.

Test APK not found

Build the test APK first:

cd android
./gradlew :app:assembleProdDebugAndroidTest

Best Practices

1. Run Robo tests first — They're free-form and catch crashes without test code
2. Always test on low-end devices — Many issues only appear on 2GB RAM devices
3. Review screenshots — Visual issues are common across device families
4. Watch videos of failures — The video shows exactly what led to the crash
5. Run on release builds — Debug builds may mask issues
6. Use AAB for Robo tests — More accurate representation of Play Store installs
7. Set --fail-fast for CI — Stop on first failure to save device-minutes
8. Archive results — Keep screenshots and videos for regression comparison