Feature/maestro smoke testing automation

.agents/skills/maestro-flow-doctor/SKILL.md

@@ -0,0 +1,41 @@
+---
+name: maestro-flow-doctor
+description: Repair a failing WooCommerce Android Maestro flow by reproducing it against the lab store, inspecting Maestro selectors, patching the smallest selector/wait/setup issue, and rerunning with repeat evidence. Human-triggered only.
+allowed-tools: Bash, Read, Edit, Grep, Glob
+user-invocable: true
+---
+
+# Maestro Flow Doctor
+
+Use this skill only for repairing existing Maestro smoke flows under `.maestro/flows/`.
+
+## Ground Rules
+
+- Always run against the lab store: pass `--store lab`.
+- Never run destructive repair loops against the shared store.
+- Never ask the user to paste credentials. Validate that `.maestro/.env.local` has the required variable names without echoing values.
+- Use Maestro MCP as the selector source of truth: `run` executes the YAML we ship, and `inspect_screen` shows the hierarchy Maestro selectors see.
+- Keep fixes minimal: selector, wait, setup, or fixture query changes only. Do not broaden coverage while repairing a flake.
+- Promotion still requires burst evidence; a local `--repeat` pass is repair evidence, not promotion.
+
+## Workflow
+
+1. Confirm the target flow path exists under `.maestro/flows/`.
+2. Run syntax and coverage checks:
+   - `.maestro/scripts/check-smoke-coverage.py`
+   - `maestro test --dry-run <flow>` if supported by the installed Maestro version; otherwise continue with a real lab run.
+3. Reproduce on a lab-store device:
+   - `.maestro/scripts/run-smoke-tests.sh --store lab --include-tags flaky_quarantine <flow>`
+4. At the failure point, use Maestro MCP `inspect_screen`.
+5. Compare the failing selector with the hierarchy:
+   - Prefer `id:` selectors exposed through `testTag`.
+   - Use generated `strings.env` values for text assertions.
+   - Do not add `point:` selectors unless the flow comment explains why no semantic selector exists.
+6. Patch the smallest file set.
+7. Rerun the single flow with repeat evidence:
+   - `.maestro/scripts/run-smoke-tests.sh --store lab --repeat 3 <flow>`
+8. Summarize:
+   - root cause,
+   - files changed,
+   - repeat result,
+   - whether the flow remains quarantined.

.agents/skills/smoke-tests/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: smoke-tests
+description: Prepare the local environment for the Maestro smoke-test suite — verify tooling, select a device, collect the APK if needed, validate the .env file by variable name only, then hand the user the exact lab-store CLI command to run or run it on request.
+allowed-tools: Bash, Read, Edit, Write, Grep, Glob, AskUserQuestion
+user-invocable: true
+---
+
+# Prepare & launch the Maestro smoke-test suite
+
+This skill is a **setup + handoff** flow. Its job is to get everything ready so `.maestro/scripts/run-smoke-tests.sh` will work on the first try, then give the user the CLI command.
+
+It does NOT own the test-runner mechanics. Ordering, per-flow recording, report generation, and the "recordings are kept only for failures, outside the repo" contract all live in the script itself.
+
+## Scope (what this skill is responsible for)
+
+1. **Tooling** — confirm `maestro` and `adb` are on PATH. If not, tell the user how to install them and stop.
+2. **Device** — confirm at least one Android device/emulator is attached (`adb devices`). If several are attached, choose the runner `--device` value with the user.
+3. **APK** — ask whether to use the currently installed app or install an APK; install only if requested.
+4. **`.env.local`** — confirm `.maestro/.env.local` exists and contains every variable the selected command references. List missing variable names only.
+5. **Handoff** — print the exact CLI command. If the user explicitly asks ("run it", "go ahead", etc.), invoke the script for them and stream its output.
+
+Everything below the handoff — P2 ordering, store selection, seed/cleanup, animation restore, lock handling, retry accounting, artifact policy, HTML + JUnit reports — is handled by `.maestro/scripts/run-smoke-tests.sh`.
+
+## Steps
+
+### 1. Check tooling
+
+Run `command -v maestro` and `command -v adb`. If either is missing:
+
+- maestro → `curl -fsSL "https://get.maestro.mobile.dev" | bash` (requires Java 17+).
+- adb → install Android SDK platform-tools and add to PATH.
+
+Tell the user which is missing and how to install it, then stop.
+
+### 2. Ensure an emulator is running
+
+Run `adb devices`. Count the lines whose second column is `device`.
+
+- **0 devices:** list AVDs with `emulator -list-avds`.
+  - If none → tell the user to create one in Android Studio (AVD Manager) and stop.
+  - If exactly one → ask the user if they want to boot it, then `emulator -avd <name> -no-snapshot-save &` (backgrounded), `adb wait-for-device`, and poll `adb shell getprop sys.boot_completed` until `1` (up to ~90s).
+  - If multiple → ask which one to boot.
+- **1+ devices:** proceed.
+
+### 3. Ask whether to install an APK
+
+Default to the currently installed app. Ask for an APK only when the user wants to test a specific build.
+
+Prompt the user for one of:
+
+- An absolute or repo-relative path to an `.apk` on disk.
+- A drag-and-drop attachment (Claude Code exposes it as a temp path like `/tmp/.../file.apk`).
+- The word `build` — in which case run `./gradlew :WooCommerce:assembleWasabiDebug` in the foreground and use `WooCommerce/build/outputs/apk/wasabi/debug/WooCommerce-wasabi-debug.apk`.
+
+Validate the chosen APK:
+
+- File exists on disk.
+- Extension is `.apk`.
+- `aapt dump badging <path> | head -1` reports `package: name='com.woocommerce.android.dev'` (the wasabi build). If it reports a different package, stop and ask the user whether to continue — a non-wasabi APK will fail the `appId` check on the first flow.
+
+Install it: `adb install -r -g <apk-path>`. If the install fails with `INSTALL_FAILED_UPDATE_INCOMPATIBLE`, tell the user a previous build with a different signature is already installed and ask whether to uninstall first (`adb uninstall com.woocommerce.android.dev`). Do not uninstall without confirmation.
+
+### 4. Validate `.maestro/.env.local`
+
+For default lab-store runs, the core required vars are:
+
+- `MAESTRO_WOO_LAB_STORE_URL`
+- `MAESTRO_WOO_LAB_EMAIL`
+- `MAESTRO_WOO_LAB_PASSWORD`
+- `MAESTRO_WOO_LAB_CONSUMER_KEY`
+- `MAESTRO_WOO_LAB_CONSUMER_SECRET`
+
+Shared-store runs require the matching `MAESTRO_WOO_SHARED_*` variables and must be explicit with `--store shared`. Additional `MAESTRO_WOO_*` vars are required per-flow (for example `MAESTRO_WOO_NOT_A_WOO_STORE_URL` for `login_not_woo_store.yaml`, `MAESTRO_WOO_JN_*` for `login_no_jetpack.yaml`). The canonical list lives in `.maestro/env.example`.
+
+To build the list of actually-referenced vars, grep flow YAMLs for `\${WOO_[A-Z_]+}` under `.maestro/flows/` and check each one resolves to a non-empty value in `.maestro/.env.local`.
+
+If the file is missing or any required var is empty:
+
+1. List the missing variable names and their purpose (mirror `.maestro/env.example`).
+2. Ask the user to edit `.maestro/.env.local` directly.
+3. Do NOT proceed until every referenced var resolves. Re-read the file after the user updates it.
+
+Never commit `.env.local`. Never ask the user to paste secret values into chat. Never echo secret values back to the user.
+
+### 5. Hand off the CLI command
+
+Once steps 1–4 all pass, print the command the user should run:
+
+```
+.maestro/scripts/run-smoke-tests.sh --store lab
+```
+
+Tell the user:
+
+- The default command runs `smoke_core` only and excludes `flaky_quarantine`.
+- Use `--include-tags smoke_extended` or a single flow path for provisional repair work.
+- Use `--store shared` only for non-destructive ad-hoc runs; destructive shared-store runs are CI-only under the store lock.
+- The runner seeds fixtures, writes a manifest, and cleans exactly those IDs on exit.
+- It captures and restores device animation settings.
+- Screen recordings are kept only for failed lab-store flows; shared-store credential paths use screenshots only.
+- Artifacts (recordings, logs, HTML + JUnit report) are written OUTSIDE the repo, under `$HOME/woocommerce-maestro-output/<timestamp>/` by default. Override with `--output-dir <path>` or the `WOO_MAESTRO_OUTPUT_DIR` env var.
+- The HTML report auto-opens at the end on macOS, or can be opened manually from the path the script prints.
+
+If the user asks to run it ("go ahead", "run it", "yes please", etc.), invoke the script yourself via `Bash` and stream its output. Pass `--apk` only if you installed an APK in step 3.
+
+When the script exits, read the last line of its output (it prints `Report:` and `Result:` summary lines) and relay a one-line summary to the user plus the clickable `file://` path to the HTML report. If any flows failed, call out the first failing flow by name — it's usually the most actionable one.
+
+## Notes
+
+- The skill's job ends at handoff. Don't reinvent the test-runner behaviour here — if something about per-flow recording, ordering, or artifact location needs to change, change it in `.maestro/scripts/run-smoke-tests.sh`.
+- `.env.local` is git-ignored. Never stage or commit it, even if the user asks you to save their credentials.
+- Artifacts default to `$HOME/woocommerce-maestro-output/` (outside the repo) — the repo's `.gitignore` still excludes the legacy `.maestro/output/` path for safety.
+- Do NOT parallelize. `adb shell screenrecord` only supports one invocation per device, and Maestro runs one flow at a time against a single emulator. The script runs sequentially by design.

.buildkite/commands/run-maestro-tests.sh

@@ -0,0 +1,93 @@
+#!/bin/bash
+set -euo pipefail
+
+# Run the Maestro smoke suite through the repository runner.
+#
+# Expected CI secrets:
+#   MAESTRO_WOO_SHARED_JETPACK_STORE_URL
+#   MAESTRO_WOO_SHARED_WPCOM_EMAIL
+#   MAESTRO_WOO_SHARED_WPCOM_PASSWORD
+#   MAESTRO_WOO_SHARED_CONSUMER_KEY (when MAESTRO_SEED=true)
+#   MAESTRO_WOO_SHARED_CONSUMER_SECRET (when MAESTRO_SEED=true)
+#
+# Optional controls:
+#   MAESTRO_STORE=shared|lab
+#   MAESTRO_INCLUDE_TAGS=smoke_core,smoke_extended,destructive
+#   MAESTRO_EXCLUDE_TAGS=flaky_quarantine,pos_tablet
+#   MAESTRO_REPEAT=3
+#   MAESTRO_APK_PATH=/path/to/beta.apk
+#   MAESTRO_SEED=true
+
+if [[ "${BUILDKITE_PULL_REQUEST:-false}" != "false" ]] &&
+  .buildkite/commands/should-skip-job.sh --job-type validation; then
+  echo "Skipping Maestro tests - no relevant changes"
+  exit 0
+fi
+
+source .maestro/scripts/configure-toolchain.sh
+
+STORE="${MAESTRO_STORE:-shared}"
+INCLUDE_TAGS="${MAESTRO_INCLUDE_TAGS:-smoke_core}"
+REPEAT="${MAESTRO_REPEAT:-1}"
+OUTPUT_DIR="${MAESTRO_OUTPUT_DIR:-WooCommerce/build/outputs/maestro-smoke}"
+SEED_ARGS=()
+EXCLUDE_TAGS_ARGS=()
+if [[ "${MAESTRO_SEED:-false}" == "true" ]]; then
+  SEED_ARGS+=(--seed)
+fi
+if [[ -n "${MAESTRO_EXCLUDE_TAGS:-}" ]]; then
+  EXCLUDE_TAGS_ARGS+=(--exclude-tags "$MAESTRO_EXCLUDE_TAGS")
+fi
+
+if [[ -n "${MAESTRO_APK_PATH:-}" ]]; then
+  APK_PATH="$MAESTRO_APK_PATH"
+else
+  echo "--- Building and installing wasabi debug APK"
+  "$(dirname "${BASH_SOURCE[0]}")/restore-cache.sh"
+  ./gradlew :WooCommerce:installWasabiDebug
+  APK_PATH=""
+fi
+
+echo "--- Running Maestro smoke tests"
+set +e
+if [[ -n "$APK_PATH" ]]; then
+  .maestro/scripts/run-smoke-tests.sh \
+    --store "$STORE" \
+    --include-tags "$INCLUDE_TAGS" \
+    --repeat "$REPEAT" \
+    --output-dir "$OUTPUT_DIR" \
+    "${SEED_ARGS[@]}" \
+    "${EXCLUDE_TAGS_ARGS[@]}" \
+    --no-open \
+    --apk "$APK_PATH"
+else
+  .maestro/scripts/run-smoke-tests.sh \
+    --store "$STORE" \
+    --include-tags "$INCLUDE_TAGS" \
+    --repeat "$REPEAT" \
+    --output-dir "$OUTPUT_DIR" \
+    "${SEED_ARGS[@]}" \
+    "${EXCLUDE_TAGS_ARGS[@]}" \
+    --no-open
+fi
+MAESTRO_EXIT_STATUS=$?
+set -e
+
+echo "--- Collecting Maestro results"
+mkdir -p WooCommerce/build/buildkite-test-analytics
+LATEST_REPORT_DIR="$(find "$OUTPUT_DIR" -mindepth 1 -maxdepth 1 -type d | sort | tail -n 1)"
+if [[ -n "$LATEST_REPORT_DIR" && -f "$LATEST_REPORT_DIR/report.xml" ]]; then
+  cp "$LATEST_REPORT_DIR/report.xml" WooCommerce/build/buildkite-test-analytics/maestro-report.xml
+fi
+if [[ "${BUILDKITE:-false}" == "true" && -n "$LATEST_REPORT_DIR" && -f "$LATEST_REPORT_DIR/report.xml" ]]; then
+  python3 .maestro/scripts/annotate-run.py --junit "$LATEST_REPORT_DIR/report.xml" |
+    buildkite-agent annotate --context maestro-smoke --style info ||
+    echo "Warning: could not publish the Maestro Buildkite annotation." >&2
+fi
+
+if [[ "$MAESTRO_EXIT_STATUS" -ne 0 ]]; then
+  echo "^^^ +++"
+  echo "Maestro smoke tests were not clean. Check flaky/failure details in artifacts."
+fi
+
+exit "$MAESTRO_EXIT_STATUS"

.buildkite/pipeline.yml

@@ -117,6 +117,24 @@ steps:
         artifact_paths:
           - "**/build/instrumented-tests/**/*"
 
+      - input: Run Maestro smoke tests?
+        prompt: "Run on-demand Maestro smoke tests. Requires MAESTRO_WOO_* secrets and an Android device/emulator."
+        key: maestro_smoke_tests_triggered
+
+      - label: ":maestro: Maestro smoke tests"
+        depends_on: maestro_smoke_tests_triggered
+        command: .buildkite/commands/run-maestro-tests.sh
+        concurrency: 1
+        concurrency_group: "woocommerce-android/maestro/shared-store"
+        plugins:
+          - $CI_TOOLKIT
+          - $TEST_COLLECTOR :
+              <<: *test_collector_common_params
+              api-token-env-name: "BUILDKITE_ANALYTICS_TOKEN_MAESTRO_TESTS"
+        artifact_paths:
+          - "WooCommerce/build/outputs/maestro-smoke/**/*"
+          - "WooCommerce/build/buildkite-test-analytics/maestro-report.xml"
+
 
   - label: "🐘 Populate Gradle build cache"
     command: .buildkite/commands/gradle-cache-build.sh

.buildkite/release-pipelines/maestro-smoke.yml

@@ -0,0 +1,27 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
+---
+
+agents:
+  queue: "android"
+
+env:
+  MAESTRO_STORE: "shared"
+  MAESTRO_INCLUDE_TAGS: "smoke_core,smoke_extended,destructive"
+  MAESTRO_EXCLUDE_TAGS: "flaky_quarantine,pos_tablet"
+  MAESTRO_REPEAT: "1"
+  MAESTRO_SEED: "true"
+
+steps:
+  - label: ":maestro: Release Maestro smoke tests"
+    command: .buildkite/commands/run-maestro-tests.sh
+    concurrency: 1
+    concurrency_group: "woocommerce-android/maestro/shared-store"
+    plugins:
+      - $CI_TOOLKIT
+      - $TEST_COLLECTOR :
+          files: "WooCommerce/build/buildkite-test-analytics/*.xml"
+          format: "junit"
+          api-token-env-name: "BUILDKITE_ANALYTICS_TOKEN_MAESTRO_TESTS"
+    artifact_paths:
+      - "WooCommerce/build/outputs/maestro-smoke/**/*"
+      - "WooCommerce/build/buildkite-test-analytics/maestro-report.xml"

.buildkite/schedules/maestro-smoke-burst.yml

@@ -0,0 +1,39 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
+---
+
+# Intended Buildkite schedule:
+#   Bi-weekly Thursday night before the Monday production rollout.
+#   Branch: active release branch.
+#
+# This is deliberately not a nightly pipeline.
+
+agents:
+  queue: "android"
+
+env:
+  MAESTRO_STORE: "shared"
+  MAESTRO_INCLUDE_TAGS: "smoke_core,smoke_extended,destructive"
+  MAESTRO_EXCLUDE_TAGS: "flaky_quarantine,pos_tablet"
+  MAESTRO_REPEAT: "3"
+  MAESTRO_SEED: "true"
+
+steps:
+  - label: ":maestro: Thursday Maestro smoke burst"
+    command: .buildkite/commands/run-maestro-tests.sh
+    concurrency: 1
+    concurrency_group: "woocommerce-android/maestro/shared-store"
+    plugins:
+      - $CI_TOOLKIT
+      - $TEST_COLLECTOR :
+          files: "WooCommerce/build/buildkite-test-analytics/*.xml"
+          format: "junit"
+          api-token-env-name: "BUILDKITE_ANALYTICS_TOKEN_MAESTRO_TESTS"
+    artifact_paths:
+      - "WooCommerce/build/outputs/maestro-smoke/**/*"
+      - "WooCommerce/build/buildkite-test-analytics/maestro-report.xml"
+
+notify:
+  - slack:
+      channels:
+        - "#woo-android-notifs"
+    if: "build.state != 'passed'"

.gitignore

@@ -56,6 +56,7 @@ local.properties
 
 # Backup Files
 *.bak
+__pycache__/
 
 # Android Studio Navigation editor temp files
 .navigation/
@@ -129,3 +130,12 @@ google-upload-credentials.json
 
 .grepai/
 docs/superpowers
+
+# Maestro smoke-test local files and artifacts.
+.env.local
+.maestro/output/
+.maestro/tmp/
+.maestro/report.xml
+.maestro/*.maestro-screenshots/
+*.maestro-screenshots/
+report.xml