Kordant/tasks/android-production/20-firebase-test-lab.md

20. Firebase Test Lab Integration

meta: id: android-production-20 feature: android-production priority: P2 depends_on: [] tags: [testing, device-farm, quality]

objective:

• Integrate Firebase Test Lab for automated testing on a wide range of physical Android devices

deliverables:

• Firebase Test Lab configuration
• Robo test configuration
• Instrumentation test execution
• Test results and reports

steps:

1. Set up Firebase Test Lab:
   • Create Firebase project
   • Link to Google Play Console
   • Enable Blaze plan (pay-as-you-go)
   • Install gcloud CLI
2. Configure test matrix:
   • Select device models:
     • Pixel 6 (API 33)
     • Pixel 4 (API 30)
     • Samsung Galaxy S21 (API 31)
     • Xiaomi Redmi (API 29)
     • Low-end device (API 28, 2GB RAM)
   • Select orientations: portrait, landscape
   • Select locales: en_US, es_ES
3. Run Robo tests:
   • Upload APK/AAB
   • Configure Robo crawl (timeout, login credentials)
   • Run on device matrix
   • Review crawl map and screenshots
4. Run instrumentation tests:
   • Upload test APK
   • Configure test runner
   • Execute UI tests on device matrix
   • Collect test results and videos
5. Integrate with CI:
   • Add gcloud commands to GitHub Actions
   • Run on release builds
   • Block release on critical failures
   • Upload results as artifacts
6. Analyze results:
   • Review screenshots from all devices
   • Watch test videos for failures
   • Check performance metrics
   • Document device-specific issues
7. Fix issues:
   • Address failures on specific devices
   • Fix OEM-specific bugs
   • Optimize for low-end devices

tests:

• Device: All selected devices pass tests
• Regression: No new failures compared to previous run
• Performance: App responsive on all devices

acceptance_criteria:

• Firebase Test Lab project configured
• Test matrix with 5+ different devices
• Robo tests running on each release build
• Instrumentation tests running on device matrix
• CI integration for automated execution
• Test results archived
• Screenshots and videos from all test runs
• Device-specific issues identified and fixed
• No crashes on any tested device
• Performance acceptable on low-end device

validation:

• Run tests via gcloud → all devices complete
• Review results → screenshots show correct UI
• Watch videos → no ANRs or crashes
• Check performance metrics → within budget
• CI pipeline → Test Lab results green

notes:

• Firebase Test Lab is free for first 100 device-minutes/day (physical)
• Robo tests are great for exploratory testing without writing tests
• Physical devices are more reliable than virtual devices
• Test on low-end devices to catch performance issues