significant android work
This commit is contained in:
271
android/firebase-test-lab/README.md
Normal file
271
android/firebase-test-lab/README.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# Firebase Test Lab Integration
|
||||
|
||||
Automated testing on real physical Android devices using Firebase Test Lab.
|
||||
Ensures the Kordant Android app works correctly across a diverse device matrix
|
||||
including Pixel, Samsung, and Xiaomi devices.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
firebase-test-lab/
|
||||
├── README.md # This file
|
||||
├── test_matrix_config.yaml # Device matrix and test configuration
|
||||
├── robo_script.json # Robo crawl script (guided UI navigation)
|
||||
├── run_robo_tests.sh # Run Robo exploratory tests
|
||||
├── run_instrumentation_tests.sh # Run instrumentation (UI) tests
|
||||
├── run_all_tests.sh # Run full test suite
|
||||
└── download_results.sh # Download and analyze test results
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Google Cloud Project** with Firebase enabled
|
||||
2. **Blaze plan** (pay-as-you-go) — Test Lab is free for the first 100 device-minutes/day on physical devices
|
||||
3. **gcloud CLI** installed and authenticated
|
||||
4. **Service account** with `Firebase Test Lab Admin` role
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Install gcloud CLI (macOS)
|
||||
brew install --cask google-cloud-sdk
|
||||
|
||||
# Authenticate
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
|
||||
# Verify
|
||||
gcloud firebase test android models list
|
||||
```
|
||||
|
||||
### Firebase Project Setup
|
||||
|
||||
1. Create a Firebase project at https://console.firebase.google.com
|
||||
2. Enable the Blaze (pay-as-you-go) plan
|
||||
3. Optionally link to Google Play Console for deep Play Store integration
|
||||
4. Create a service account and download JSON key:
|
||||
- IAM & Admin → Service Accounts → Create Service Account
|
||||
- Role: `Firebase Test Lab Admin` (roles/firebase.testlab.admin)
|
||||
- Create and download JSON key
|
||||
|
||||
## Device Matrix
|
||||
|
||||
The app is tested on 5 devices across 2 orientations and 2 locales
|
||||
(20 device/locale/orientation combinations total):
|
||||
|
||||
| Device | Model ID | API | Screen | RAM | Target |
|
||||
|--------|----------|-----|--------|-----|--------|
|
||||
| Pixel 6 | `Pixel6` | 33 | 1080×2400 | 8GB | Primary target |
|
||||
| Pixel 4 | `Pixel4` | 30 | 1080×2280 | 6GB | Older device |
|
||||
| Galaxy S21 | `GalaxyS21` | 31 | 1080×2400 | 8GB | Samsung |
|
||||
| Redmi Note 8 | `RedmiNote8` | 29 | 1080×2340 | 4GB | Xiaomi / budget |
|
||||
| Aquest M2 | `AquestM2` | 28 | 720×1280 | 2GB | Low-end / minimum spec |
|
||||
|
||||
**Orientations:** portrait, landscape
|
||||
**Locales:** en_US (English US), es_ES (Spanish Spain)
|
||||
|
||||
## Running Tests
|
||||
|
||||
### 1. Build the app
|
||||
|
||||
```bash
|
||||
cd android
|
||||
./gradlew :app:assembleProdRelease :app:assembleProdDebugAndroidTest
|
||||
```
|
||||
|
||||
### 2. Run Robo Tests (exploratory crash detection)
|
||||
|
||||
Robo tests automatically crawl the app UI without requiring any test code.
|
||||
They detect crashes, ANRs, and UI rendering issues.
|
||||
|
||||
```bash
|
||||
cd android/firebase-test-lab
|
||||
./run_robo_tests.sh --project-id kordant-android
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--project-id` — Firebase project ID (default: `kordant-android`)
|
||||
- `--app-aab` — Path to AAB (preferred, more accurate)
|
||||
- `--app-apk` — Path to APK (fallback)
|
||||
- `--robo-script` — Path to robo crawl script
|
||||
- `--timeout` — Max crawl time in seconds (default: 600)
|
||||
- `--dry-run` — Preview command without executing
|
||||
|
||||
### 3. Run Instrumentation Tests (UI tests with assertions)
|
||||
|
||||
Runs the existing UI test suite (AuthFlowTest, DashboardUITest, ServiceUITests, etc.)
|
||||
across the full device matrix.
|
||||
|
||||
```bash
|
||||
cd android/firebase-test-lab
|
||||
./run_instrumentation_tests.sh --project-id kordant-android
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--project-id` — Firebase project ID
|
||||
- `--app-apk` — Path to app APK (auto-detected)
|
||||
- `--test-apk` — Path to test APK (auto-detected)
|
||||
- `--dry-run` — Preview command without executing
|
||||
|
||||
### 4. Run Full Test Suite
|
||||
|
||||
```bash
|
||||
cd android/firebase-test-lab
|
||||
./run_all_tests.sh --project-id kordant-android
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--skip-build` — Skip Gradle build step
|
||||
- `--skip-robo` — Skip Robo tests
|
||||
- `--skip-instr` — Skip instrumentation tests
|
||||
- `--dry-run` — Preview commands without executing
|
||||
|
||||
### 5. Download Results
|
||||
|
||||
```bash
|
||||
cd android/firebase-test-lab
|
||||
./download_results.sh --project-id kordant-android --download-all
|
||||
```
|
||||
|
||||
This downloads:
|
||||
- Test result XMLs (JUnit format)
|
||||
- Screenshots (PNG)
|
||||
- Test videos (MP4)
|
||||
- Performance metrics (JSON)
|
||||
- Logcat output
|
||||
- Crawl maps (Robo test UI exploration paths)
|
||||
|
||||
## CI Integration
|
||||
|
||||
The GitHub Actions workflow at `.github/workflows/firebase-test-lab.yml`
|
||||
runs automatically on pushes and PRs that modify Android code.
|
||||
|
||||
### CI Pipeline Flow
|
||||
|
||||
1. **Build job** — Compiles release and test APKs
|
||||
2. **Robo Tests job** — Runs crash/ANR detection on all 20 device configurations
|
||||
3. **Instrumentation Tests job** — Runs UI test suite on all 20 device configurations
|
||||
4. **Test Summary job** — Collects results and determines pass/fail
|
||||
|
||||
### GitHub Secrets Required
|
||||
|
||||
| Secret | Description |
|
||||
|--------|-------------|
|
||||
| `GCP_SA_KEY_TEST_LAB` | Service account JSON key with Test Lab admin role |
|
||||
| `FIREBASE_PROJECT_ID` | Firebase project ID (default: `kordant-android`) |
|
||||
|
||||
### Adding to CI
|
||||
|
||||
The workflow triggers on:
|
||||
- Push to `main` with Android changes
|
||||
- PR to `main` with Android changes
|
||||
- Manual trigger via `workflow_dispatch`
|
||||
|
||||
To block release builds on test failures, add the test-summary job as a
|
||||
required check in your branch protection rules.
|
||||
|
||||
## Robo Test Script
|
||||
|
||||
The `robo_script.json` file guides the Robo crawler through the app's
|
||||
critical user flow:
|
||||
|
||||
1. Wait for splash screen
|
||||
2. Click "Get Started" on the onboarding screen
|
||||
3. Click "Sign In" on the login screen
|
||||
4. Enter email and password
|
||||
5. Submit sign-in
|
||||
6. Navigate through: Dashboard → Services → Alerts → Settings
|
||||
|
||||
This ensures the crawler reaches authenticated screens. The test user
|
||||
credentials are injected via `${ROBO_ID}` for unique user per test run.
|
||||
|
||||
## Test Accounts
|
||||
|
||||
Robo tests support test accounts for automatic login during the crawl.
|
||||
Configure credentials securely:
|
||||
|
||||
```bash
|
||||
gcloud firebase test android run \
|
||||
--type robo \
|
||||
--app app.apk \
|
||||
--device model=Pixel6,version=33,locale=en_US,orientation=portrait \
|
||||
--test-accounts username=test@kordant.com,password=$ROBO_PASSWORD
|
||||
```
|
||||
|
||||
For CI, store credentials in GitHub Secrets and pass as environment variables.
|
||||
|
||||
## Analyzing Results
|
||||
|
||||
### In Firebase Console
|
||||
|
||||
1. Navigate to https://console.firebase.google.com/project/YOUR_PROJECT/testlab
|
||||
2. View test matrices grouped by history name
|
||||
3. Click on a matrix to see per-device results
|
||||
4. Watch test videos to identify UI issues
|
||||
5. Review screenshots for visual regressions
|
||||
6. Check performance metrics for responsiveness
|
||||
|
||||
### Performance Budget
|
||||
|
||||
| Metric | Target | Device |
|
||||
|--------|--------|--------|
|
||||
| Cold start | < 1500ms | Pixel 6 |
|
||||
| Warm start | < 1000ms | Pixel 6 |
|
||||
| Robo crawl | Complete in < 10min per device | All |
|
||||
| No crashes | 0 crashes | All |
|
||||
| No ANRs | 0 ANRs | All |
|
||||
|
||||
### Device-Specific Issues to Watch
|
||||
|
||||
- **Low-end devices (API 28, 2GB RAM):** Check for OOM, slow rendering, lag
|
||||
- **Xiaomi:** Check for MIUI-specific permission quirks
|
||||
- **Samsung:** Check for One UI theme compatibility
|
||||
- **Landscape:** Verify proper layout adaptation
|
||||
- **Spanish locale:** Check for text truncation or layout overflow
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Permission denied" when running scripts
|
||||
|
||||
```bash
|
||||
chmod +x android/firebase-test-lab/*.sh
|
||||
```
|
||||
|
||||
### "No authenticated account" error
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
### "Project not found" error
|
||||
|
||||
Verify the project exists and has the Blaze plan enabled:
|
||||
```bash
|
||||
gcloud projects list
|
||||
gcloud firebase test android models list --project YOUR_PROJECT_ID
|
||||
```
|
||||
|
||||
### "Quota exceeded" error
|
||||
|
||||
Firebase Test Lab has daily quotas. Check usage in the Firebase Console.
|
||||
The free tier provides 100 device-minutes/day on physical devices.
|
||||
|
||||
### Test APK not found
|
||||
|
||||
Build the test APK first:
|
||||
```bash
|
||||
cd android
|
||||
./gradlew :app:assembleProdDebugAndroidTest
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Run Robo tests first** — They're free-form and catch crashes without test code
|
||||
2. **Always test on low-end devices** — Many issues only appear on 2GB RAM devices
|
||||
3. **Review screenshots** — Visual issues are common across device families
|
||||
4. **Watch videos of failures** — The video shows exactly what led to the crash
|
||||
5. **Run on release builds** — Debug builds may mask issues
|
||||
6. **Use AAB for Robo tests** — More accurate representation of Play Store installs
|
||||
7. **Set --fail-fast for CI** — Stop on first failure to save device-minutes
|
||||
8. **Archive results** — Keep screenshots and videos for regression comparison
|
||||
225
android/firebase-test-lab/download_results.sh
Executable file
225
android/firebase-test-lab/download_results.sh
Executable file
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# Download Firebase Test Lab Results
|
||||
# =============================================================================
|
||||
# Downloads and organizes test results from Firebase Test Lab, including
|
||||
# screenshots, videos, performance metrics, and test reports.
|
||||
#
|
||||
# Usage:
|
||||
# ./download_results.sh [options]
|
||||
#
|
||||
# Options:
|
||||
# --project-id Firebase project ID (default: kordant-android)
|
||||
# --matrix-id Specific matrix ID to download (optional, downloads latest)
|
||||
# --output-dir Output directory (default: ./test_results)
|
||||
# --download-all Download all artifacts including screenshots and videos
|
||||
# --help Show this help message
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Default values
|
||||
PROJECT_ID="${FIREBASE_PROJECT_ID:-kordant-android}"
|
||||
MATRIX_ID=""
|
||||
OUTPUT_DIR="${SCRIPT_DIR}/test_results"
|
||||
DOWNLOAD_ALL=false
|
||||
|
||||
# ============================================================
|
||||
# Parse arguments
|
||||
# ============================================================
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--project-id)
|
||||
PROJECT_ID="$2"
|
||||
shift 2
|
||||
;;
|
||||
--matrix-id)
|
||||
MATRIX_ID="$2"
|
||||
shift 2
|
||||
;;
|
||||
--output-dir)
|
||||
OUTPUT_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--download-all)
|
||||
DOWNLOAD_ALL=true
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
grep "^#" "$0" | grep -v "^#!/" | sed 's/^# //'
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Usage: $0 --help"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ============================================================
|
||||
# Validate gcloud
|
||||
# ============================================================
|
||||
if ! command -v gcloud &> /dev/null; then
|
||||
echo "Error: gcloud CLI is not installed."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Find the GCS bucket for test results
|
||||
# ============================================================
|
||||
echo "🔍 Finding Firebase Test Lab results bucket..."
|
||||
echo "Project ID: $PROJECT_ID"
|
||||
|
||||
# Get the storage bucket name from the Firebase project
|
||||
# Test Lab results are stored in gs://<project-id>-test-lab-<random-suffix>
|
||||
RESULTS_BUCKET=$(gsutil ls 2>/dev/null | grep "${PROJECT_ID}-test-lab-" || echo "")
|
||||
|
||||
if [ -z "$RESULTS_BUCKET" ]; then
|
||||
echo "No test lab bucket found via gsutil. Trying gcloud to list recent matrices..."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# List recent test matrices
|
||||
# ============================================================
|
||||
echo "📋 Recent test matrices:"
|
||||
echo ""
|
||||
|
||||
RECENT_MATRICES=$(gcloud firebase test android matrices list \
|
||||
--project "$PROJECT_ID" \
|
||||
--limit 10 \
|
||||
--format="table(matrixId, state, gcsPath, createTime)" 2>/dev/null || echo "No matrices found.")
|
||||
|
||||
echo "$RECENT_MATRICES"
|
||||
echo ""
|
||||
|
||||
# ============================================================
|
||||
# If no matrix ID specified, get the latest
|
||||
# ============================================================
|
||||
if [ -z "$MATRIX_ID" ]; then
|
||||
MATRIX_ID=$(gcloud firebase test android matrices list \
|
||||
--project "$PROJECT_ID" \
|
||||
--limit 1 \
|
||||
--format="value(matrixId)" 2>/dev/null || echo "")
|
||||
fi
|
||||
|
||||
if [ -z "$MATRIX_ID" ]; then
|
||||
echo "No test matrices found. Run tests first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Selected matrix: $MATRIX_ID"
|
||||
echo ""
|
||||
|
||||
# ============================================================
|
||||
# Get GCS path for this matrix
|
||||
# ============================================================
|
||||
MATRIX_INFO=$(gcloud firebase test android matrices describe "$MATRIX_ID" \
|
||||
--project "$PROJECT_ID" \
|
||||
--format="json" 2>/dev/null || echo "{}")
|
||||
|
||||
GCS_PATH=$(echo "$MATRIX_INFO" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('gcsPath',''))" 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$GCS_PATH" ]; then
|
||||
echo "Error: Could not determine GCS path for matrix $MATRIX_ID"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "GCS Path: $GCS_PATH"
|
||||
echo ""
|
||||
|
||||
# ============================================================
|
||||
# Create output directory
|
||||
# ============================================================
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
# ============================================================
|
||||
# Download results summary (JUnit XML)
|
||||
# ============================================================
|
||||
echo "📄 Downloading test results summary..."
|
||||
echo " Output: $OUTPUT_DIR/"
|
||||
|
||||
# Download the test_results.xml (JUnit format)
|
||||
gsutil -m cp "$GCS_PATH/**/test_result.xml" "$OUTPUT_DIR/" 2>/dev/null || true
|
||||
gsutil -m cp "$GCS_PATH/**/test_results.xml" "$OUTPUT_DIR/" 2>/dev/null || true
|
||||
|
||||
# Download the performance metrics
|
||||
gsutil -m cp "$GCS_PATH/**/performance_metrics.json" "$OUTPUT_DIR/performance/" 2>/dev/null || true
|
||||
|
||||
# Download the logcat output (if available)
|
||||
gsutil -m cp "$GCS_PATH/**/logcat" "$OUTPUT_DIR/logcat/" 2>/dev/null || true
|
||||
|
||||
# ============================================================
|
||||
# Download screenshots and videos (if --download-all)
|
||||
# ============================================================
|
||||
if [ "$DOWNLOAD_ALL" = true ]; then
|
||||
echo ""
|
||||
echo "🖼️ Downloading screenshots and videos..."
|
||||
|
||||
# Download all PNG screenshots
|
||||
mkdir -p "$OUTPUT_DIR/screenshots"
|
||||
gsutil -m cp "$GCS_PATH/**/*.png" "$OUTPUT_DIR/screenshots/" 2>/dev/null || true
|
||||
SCREENSHOT_COUNT=$(find "$OUTPUT_DIR/screenshots" -name "*.png" 2>/dev/null | wc -l | tr -d ' ')
|
||||
echo " Screenshots downloaded: $SCREENSHOT_COUNT"
|
||||
|
||||
# Download all MP4 videos
|
||||
mkdir -p "$OUTPUT_DIR/videos"
|
||||
gsutil -m cp "$GCS_PATH/**/*.mp4" "$OUTPUT_DIR/videos/" 2>/dev/null || true
|
||||
VIDEO_COUNT=$(find "$OUTPUT_DIR/videos" -name "*.mp4" 2>/dev/null | wc -l | tr -d ' ')
|
||||
echo " Videos downloaded: $VIDEO_COUNT"
|
||||
|
||||
# Download crawl maps (Robo test output)
|
||||
mkdir -p "$OUTPUT_DIR/crawl_maps"
|
||||
gsutil -m cp "$GCS_PATH/**/*.json" "$OUTPUT_DIR/crawl_maps/" 2>/dev/null || true
|
||||
CRAWL_COUNT=$(find "$OUTPUT_DIR/crawl_maps" -name "*.json" 2>/dev/null | wc -l | tr -d ' ')
|
||||
echo " Crawl maps downloaded: $CRAWL_COUNT"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Generate report
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "📊 Results Summary"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Parse matrix info for summary
|
||||
MATRIX_STATE=$(echo "$MATRIX_INFO" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('state','UNKNOWN'))" 2>/dev/null || echo "UNKNOWN")
|
||||
echo "Matrix State: $MATRIX_STATE"
|
||||
echo ""
|
||||
|
||||
# Show outcome summary per device
|
||||
echo "$MATRIX_INFO" | python3 -c "
|
||||
import sys, json
|
||||
d = json.load(sys.stdin)
|
||||
tests = d.get('testExecutions', [])
|
||||
for t in tests:
|
||||
device = t.get('device', {})
|
||||
model = device.get('androidModelId', '?')
|
||||
version = device.get('androidVersionId', '?')
|
||||
state = t.get('state', '?')
|
||||
outcome = t.get('outcome', {}).get('summary', '?')
|
||||
print(f' {model} (API {version}): {state} - {outcome}')
|
||||
" 2>/dev/null || echo " (Could not parse individual device results)"
|
||||
|
||||
echo ""
|
||||
echo "Output directory: $OUTPUT_DIR"
|
||||
echo ""
|
||||
echo "View in Firebase Console:"
|
||||
echo " https://console.firebase.google.com/project/$PROJECT_ID/testlab/histories"
|
||||
|
||||
# ============================================================
|
||||
# Check for failures
|
||||
# ============================================================
|
||||
if echo "$MATRIX_STATE" | grep -qi "fail\|error\|invalid"; then
|
||||
echo ""
|
||||
echo "⚠️ Test matrix has failures! Review the results."
|
||||
exit 1
|
||||
else
|
||||
echo ""
|
||||
echo "✅ Test matrix completed successfully!"
|
||||
exit 0
|
||||
fi
|
||||
247
android/firebase-test-lab/run_all_tests.sh
Executable file
247
android/firebase-test-lab/run_all_tests.sh
Executable file
@@ -0,0 +1,247 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# Run All Firebase Test Lab Tests
|
||||
# =============================================================================
|
||||
# Orchestrates the full Firebase Test Lab test suite:
|
||||
# 1. Robo exploratory tests (crash detection without test code)
|
||||
# 2. Instrumentation tests (UI tests with assertions)
|
||||
#
|
||||
# This script builds the app, runs both test types sequentially, and
|
||||
# reports results.
|
||||
#
|
||||
# Prerequisites:
|
||||
# 1. gcloud CLI installed and authenticated
|
||||
# 2. Firebase project with Blaze plan enabled
|
||||
# 3. Service account with Firebase Test Lab admin role
|
||||
# 4. Java 17+ for Android builds
|
||||
#
|
||||
# Usage:
|
||||
# ./run_all_tests.sh [options]
|
||||
#
|
||||
# Options:
|
||||
# --project-id Firebase project ID (default: kordant-android)
|
||||
# --skip-build Skip the Gradle build step
|
||||
# --skip-robo Skip Robo tests (run instrumentation only)
|
||||
# --skip-instr Skip instrumentation tests (run Robo only)
|
||||
# --dry-run Print commands without executing
|
||||
# --help Show this help message
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
# Default values
|
||||
PROJECT_ID="${FIREBASE_PROJECT_ID:-kordant-android}"
|
||||
SKIP_BUILD=false
|
||||
SKIP_ROBO=false
|
||||
SKIP_INSTR=false
|
||||
DRY_RUN=false
|
||||
|
||||
# ============================================================
|
||||
# Parse arguments
|
||||
# ============================================================
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--project-id)
|
||||
PROJECT_ID="$2"
|
||||
shift 2
|
||||
;;
|
||||
--skip-build)
|
||||
SKIP_BUILD=true
|
||||
shift
|
||||
;;
|
||||
--skip-robo)
|
||||
SKIP_ROBO=true
|
||||
shift
|
||||
;;
|
||||
--skip-instr)
|
||||
SKIP_INSTR=true
|
||||
shift
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=true
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
grep "^#" "$0" | grep -v "^#!/" | sed 's/^# //'
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Usage: $0 --help"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ============================================================
|
||||
# Timestamp helper
|
||||
# ============================================================
|
||||
timestamp() {
|
||||
date "+%Y-%m-%d %H:%M:%S"
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# Validate prerequisites
|
||||
# ============================================================
|
||||
echo "=========================================="
|
||||
echo "Firebase Test Lab - Full Test Suite"
|
||||
echo "=========================================="
|
||||
echo "Started at: $(timestamp)"
|
||||
echo "Project ID: $PROJECT_ID"
|
||||
echo ""
|
||||
|
||||
# Check gcloud
|
||||
if ! command -v gcloud &> /dev/null; then
|
||||
echo "Error: gcloud CLI is not installed."
|
||||
echo "Install it from: https://cloud.google.com/sdk/docs/install"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check authentication
|
||||
if ! gcloud auth application-default print-access-token &> /dev/null; then
|
||||
echo "Warning: Application default credentials not set."
|
||||
echo "Run: gcloud auth application-default login"
|
||||
echo ""
|
||||
|
||||
# Check if user is authenticated at all
|
||||
if ! gcloud auth list --filter=status:ACTIVE --format="value(account)" &> /dev/null; then
|
||||
echo "Error: No active gcloud account. Run: gcloud auth login"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Step 1: Build the app (if not skipped)
|
||||
# ============================================================
|
||||
if [ "$SKIP_BUILD" = false ]; then
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "📦 Step 1: Building Android APKs"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
cd "$PROJECT_DIR/android"
|
||||
|
||||
# Determine Java version
|
||||
JAVA_VERSION=$(java -version 2>&1 | head -1 | grep -oP 'version "\K[^"]+' || echo "unknown")
|
||||
echo "Java version: $JAVA_VERSION"
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "[DRY-RUN] Would run: ./gradlew :app:assembleProdRelease :app:assembleProdDebugAndroidTest"
|
||||
else
|
||||
echo "Building release APK and test APK..."
|
||||
./gradlew :app:assembleProdRelease :app:assembleProdDebugAndroidTest
|
||||
|
||||
echo ""
|
||||
echo "Build completed. APK locations:"
|
||||
find "$PROJECT_DIR/android/app/build/outputs" -name "*.apk" -type f 2>/dev/null | while read -r apk; do
|
||||
size=$(stat -f%z "$apk" 2>/dev/null || stat -c%s "$apk" 2>/dev/null || echo "?")
|
||||
echo " $(basename "$apk") ($(echo "scale=1; $size/1048576" | bc) MB)"
|
||||
done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
else
|
||||
echo "⏭️ Build step skipped."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Step 2: Run Robo tests
|
||||
# ============================================================
|
||||
ROBO_RESULT=0
|
||||
if [ "$SKIP_ROBO" = false ]; then
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "🤖 Step 2: Running Robo Tests"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
ROBO_SCRIPT="${SCRIPT_DIR}/run_robo_tests.sh"
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "[DRY-RUN] Would run: $ROBO_SCRIPT --project-id $PROJECT_ID"
|
||||
else
|
||||
if [ -f "$ROBO_SCRIPT" ]; then
|
||||
bash "$ROBO_SCRIPT" --project-id "$PROJECT_ID" || ROBO_RESULT=$?
|
||||
else
|
||||
echo "Warning: $ROBO_SCRIPT not found, skipping Robo tests."
|
||||
ROBO_RESULT=0
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
else
|
||||
echo "⏭️ Robo tests skipped."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Step 3: Run instrumentation tests
|
||||
# ============================================================
|
||||
INSTR_RESULT=0
|
||||
if [ "$SKIP_INSTR" = false ]; then
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "🧪 Step 3: Running Instrumentation Tests"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
INSTR_SCRIPT="${SCRIPT_DIR}/run_instrumentation_tests.sh"
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "[DRY-RUN] Would run: $INSTR_SCRIPT --project-id $PROJECT_ID"
|
||||
else
|
||||
if [ -f "$INSTR_SCRIPT" ]; then
|
||||
bash "$INSTR_SCRIPT" --project-id "$PROJECT_ID" || INSTR_RESULT=$?
|
||||
else
|
||||
echo "Warning: $INSTR_SCRIPT not found, skipping instrumentation tests."
|
||||
INSTR_RESULT=0
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
else
|
||||
echo "⏭️ Instrumentation tests skipped."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Step 4: Summary
|
||||
# ============================================================
|
||||
echo "=========================================="
|
||||
echo "📊 Test Suite Summary"
|
||||
echo "=========================================="
|
||||
echo "Finished at: $(timestamp)"
|
||||
echo ""
|
||||
|
||||
if [ "$SKIP_ROBO" = false ]; then
|
||||
if [ $ROBO_RESULT -eq 0 ]; then
|
||||
echo "✅ Robo Tests: PASSED"
|
||||
else
|
||||
echo "❌ Robo Tests: FAILED (exit code $ROBO_RESULT)"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$SKIP_INSTR" = false ]; then
|
||||
if [ $INSTR_RESULT -eq 0 ]; then
|
||||
echo "✅ Instrumentation: PASSED"
|
||||
else
|
||||
echo "❌ Instrumentation: FAILED (exit code $INSTR_RESULT)"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "View all results in Firebase Console:"
|
||||
echo " https://console.firebase.google.com/project/$PROJECT_ID/testlab"
|
||||
echo ""
|
||||
|
||||
# Determine overall exit code
|
||||
if [ "$SKIP_ROBO" = false ] && [ $ROBO_RESULT -ne 0 ]; then
|
||||
exit $ROBO_RESULT
|
||||
fi
|
||||
if [ "$SKIP_INSTR" = false ] && [ $INSTR_RESULT -ne 0 ]; then
|
||||
exit $INSTR_RESULT
|
||||
fi
|
||||
|
||||
exit 0
|
||||
239
android/firebase-test-lab/run_robo_tests.sh
Executable file
239
android/firebase-test-lab/run_robo_tests.sh
Executable file
@@ -0,0 +1,239 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# Run Robo Tests on Firebase Test Lab
|
||||
# =============================================================================
|
||||
# This script runs Robo exploratory tests on Firebase Test Lab across the
|
||||
# configured device matrix. Robo tests automatically crawl the app UI to
|
||||
# find crashes and ANRs without requiring instrumented test code.
|
||||
#
|
||||
# Prerequisites:
|
||||
# 1. gcloud CLI installed and authenticated (gcloud auth login)
|
||||
# 2. Firebase project created and Blaze plan enabled
|
||||
# 3. Google Play Console linked to Firebase project (optional, for deep links)
|
||||
# 4. Service account with Firebase Test Lab admin role
|
||||
#
|
||||
# Usage:
|
||||
# ./run_robo_tests.sh [options]
|
||||
#
|
||||
# Options:
|
||||
# --project-id Firebase project ID (default: kordant-android)
|
||||
# --app-aab Path to app AAB (default: auto-detected)
|
||||
# --app-apk Path to app APK (default: auto-detected)
|
||||
# --robo-script Path to robo script JSON (default: robo_script.json)
|
||||
# --timeout Max robo crawl time in seconds (default: 600)
|
||||
# --dry-run Print gcloud command without executing
|
||||
# --help Show this help message
|
||||
#
|
||||
# Reference: https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
# Default values
|
||||
PROJECT_ID="${FIREBASE_PROJECT_ID:-kordant-android}"
|
||||
APP_PATH=""
|
||||
ROBO_SCRIPT="${SCRIPT_DIR}/robo_script.json"
|
||||
MAX_CRAWL_TIME=600
|
||||
DRY_RUN=false
|
||||
|
||||
# Device matrix from test_matrix_config.yaml (generated via gcloud --device flags)
|
||||
# Each device runs with each orientation and locale combination
|
||||
declare -a DEVICE_ARGS=(
|
||||
# Pixel 6 - Primary target device (API 33)
|
||||
"--device model=Pixel6,version=33,locale=en_US,orientation=portrait"
|
||||
"--device model=Pixel6,version=33,locale=en_US,orientation=landscape"
|
||||
"--device model=Pixel6,version=33,locale=es_ES,orientation=portrait"
|
||||
"--device model=Pixel6,version=33,locale=es_ES,orientation=landscape"
|
||||
|
||||
# Pixel 4 - Older Pixel device (API 30)
|
||||
"--device model=Pixel4,version=30,locale=en_US,orientation=portrait"
|
||||
"--device model=Pixel4,version=30,locale=en_US,orientation=landscape"
|
||||
"--device model=Pixel4,version=30,locale=es_ES,orientation=portrait"
|
||||
"--device model=Pixel4,version=30,locale=es_ES,orientation=landscape"
|
||||
|
||||
# Samsung Galaxy S21 - Popular Samsung device (API 31)
|
||||
"--device model=GalaxyS21,version=31,locale=en_US,orientation=portrait"
|
||||
"--device model=GalaxyS21,version=31,locale=en_US,orientation=landscape"
|
||||
"--device model=GalaxyS21,version=31,locale=es_ES,orientation=portrait"
|
||||
"--device model=GalaxyS21,version=31,locale=es_ES,orientation=landscape"
|
||||
|
||||
# Xiaomi Redmi Note 8 - Budget device (API 29)
|
||||
"--device model=RedmiNote8,version=29,locale=en_US,orientation=portrait"
|
||||
"--device model=RedmiNote8,version=29,locale=en_US,orientation=landscape"
|
||||
"--device model=RedmiNote8,version=29,locale=es_ES,orientation=portrait"
|
||||
"--device model=RedmiNote8,version=29,locale=es_ES,orientation=landscape"
|
||||
|
||||
# Low-end device - Minimum spec target (API 28, 2GB RAM equivalent)
|
||||
"--device model=AquestM2,version=28,locale=en_US,orientation=portrait"
|
||||
"--device model=AquestM2,version=28,locale=en_US,orientation=landscape"
|
||||
"--device model=AquestM2,version=28,locale=es_ES,orientation=portrait"
|
||||
"--device model=AquestM2,version=28,locale=es_ES,orientation=landscape"
|
||||
)
|
||||
|
||||
# ============================================================
|
||||
# Helper: Print usage
|
||||
# ============================================================
|
||||
usage() {
|
||||
grep "^#" "$0" | grep -v "^#!/" | sed 's/^# //'
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# Parse arguments
|
||||
# ============================================================
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--project-id)
|
||||
PROJECT_ID="$2"
|
||||
shift 2
|
||||
;;
|
||||
--app-aab)
|
||||
APP_PATH="$2"
|
||||
shift 2
|
||||
;;
|
||||
--app-apk)
|
||||
APP_PATH="$2"
|
||||
shift 2
|
||||
;;
|
||||
--robo-script)
|
||||
ROBO_SCRIPT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--timeout)
|
||||
MAX_CRAWL_TIME="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=true
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Usage: $0 --help"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ============================================================
|
||||
# Auto-detect APK/AAB path if not provided
|
||||
# ============================================================
|
||||
if [ -z "$APP_PATH" ]; then
|
||||
# Prefer AAB for more accurate Play Store representation
|
||||
APP_PATH=$(find "$PROJECT_DIR/android/app/build/outputs/bundle" -name "*-release.aab" 2>/dev/null | head -1)
|
||||
|
||||
# Fall back to APK if AAB not found
|
||||
if [ -z "$APP_PATH" ]; then
|
||||
APP_PATH=$(find "$PROJECT_DIR/android/app/build/outputs/apk" -name "*-release.apk" 2>/dev/null | head -1)
|
||||
fi
|
||||
|
||||
# Last resort: any APK
|
||||
if [ -z "$APP_PATH" ]; then
|
||||
APP_PATH=$(find "$PROJECT_DIR/android/app/build/outputs/apk" -name "*.apk" 2>/dev/null | head -1)
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$APP_PATH" ]; then
|
||||
echo "Error: Could not find app APK or AAB."
|
||||
echo ""
|
||||
echo "Build the app first:"
|
||||
echo " ./gradlew :app:assembleProdRelease"
|
||||
echo " # or for AAB:"
|
||||
echo " ./gradlew :app:bundleProdRelease"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$ROBO_SCRIPT" ]; then
|
||||
echo "Warning: Robo script not found at $ROBO_SCRIPT"
|
||||
echo "Robo will run without guided script (fully autonomous crawl)."
|
||||
ROBO_SCRIPT=""
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Determine type flag based on file extension
|
||||
# ============================================================
|
||||
if [[ "$APP_PATH" == *.aab ]]; then
|
||||
TYPE_FLAG="--type robo"
|
||||
APP_FLAG="--app-package com.kordant.android"
|
||||
AAB_FLAG="--aab \"$APP_PATH\""
|
||||
APK_FLAG=""
|
||||
else
|
||||
TYPE_FLAG="--type robo"
|
||||
APP_FLAG=""
|
||||
AAB_FLAG=""
|
||||
APK_FLAG="--app \"$APP_PATH\""
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Print configuration
|
||||
# ============================================================
|
||||
echo "=========================================="
|
||||
echo "Firebase Test Lab - Robo Tests"
|
||||
echo "=========================================="
|
||||
echo "Project ID: $PROJECT_ID"
|
||||
echo "App: $APP_PATH"
|
||||
echo "Robo Script: ${ROBO_SCRIPT:-none (fully autonomous)}"
|
||||
echo "Max Crawl Time: ${MAX_CRAWL_TIME}s"
|
||||
echo "Devices: ${#DEVICE_ARGS[@]} configurations"
|
||||
echo ""
|
||||
|
||||
# ============================================================
|
||||
# Build gcloud command
|
||||
# ============================================================
|
||||
GCLOUD_CMD="gcloud firebase test android run \
|
||||
$TYPE_FLAG \
|
||||
--project \"$PROJECT_ID\" \
|
||||
$APP_FLAG \
|
||||
$AAB_FLAG \
|
||||
$APK_FLAG \
|
||||
--timeout 60m \
|
||||
--max-crawl-time $MAX_CRAWL_TIME \
|
||||
--record-video \
|
||||
--performance-metrics \
|
||||
--results-history-name \"Kordant Android Robo Tests\""
|
||||
|
||||
# Add robo script if available
|
||||
if [ -n "$ROBO_SCRIPT" ]; then
|
||||
GCLOUD_CMD="$GCLOUD_CMD --robo-script \"$ROBO_SCRIPT\""
|
||||
fi
|
||||
|
||||
# Add device configurations
|
||||
for device in "${DEVICE_ARGS[@]}"; do
|
||||
GCLOUD_CMD="$GCLOUD_CMD $device"
|
||||
done
|
||||
|
||||
echo "Command:"
|
||||
echo "$GCLOUD_CMD"
|
||||
echo ""
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "DRY RUN - Command not executed."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# Execute
|
||||
# ============================================================
|
||||
echo "Starting Robo tests..."
|
||||
echo ""
|
||||
eval "$GCLOUD_CMD"
|
||||
|
||||
EXIT_CODE=$?
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo ""
|
||||
echo "✅ Robo tests completed successfully!"
|
||||
echo "View results in Firebase Console: https://console.firebase.google.com/project/$PROJECT_ID/testlab"
|
||||
echo "Review crawl maps, screenshots, and videos for each device."
|
||||
else
|
||||
echo ""
|
||||
echo "❌ Robo tests failed with exit code $EXIT_CODE"
|
||||
echo "View results in Firebase Console: https://console.firebase.google.com/project/$PROJECT_ID/testlab"
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
Reference in New Issue
Block a user