feat: Percy feature testbed — one-command build seeder (Units 1–10)

.npmrc

@@ -0,0 +1,11 @@
+# .npmrc — supply-chain hardening
+# Required by BrowserStack Supply Chain Security audit (PER-10291 / SC-12282, Enigma).
+# Tech spec: ENG > Supply Chain Security Enhancements.
+
+ignore-scripts=true      # block malicious dependency lifecycle scripts
+strict-ssl=true          # enforce TLS to the registry
+save-exact=true          # pin exact versions (no caret ranges)
+audit-level=high         # fail on high+ severity advisories
+engine-strict=true       # refuse incompatible Node engines
+legacy-peer-deps=false   # preserve npm 7+ peer-dependency resolution
+access=restricted        # private-package default (private repo)

README.md

@@ -1,80 +1,182 @@
 # percy-feature-testbed
 
-One command that populates a Percy environment with **real renderer-backed** builds
-exercising **every Percy feature** — for manual QA. A human then logs into the
-pre-populated accounts and inspects each feature in the dashboard.
+One command that seeds a Percy environment with **real builds of every kind** —
+baseline / changed / new / removed, visual-git / A-B, recurring diff, AI, regions,
+approval automation — so QA can log in and inspect each feature in the dashboard.
 
+The flow, proven end-to-end against Percy: **create a project → fetch its token →
+create builds** — `percy snapshot` serves generated fixture pages and renders them in a
+real browser, so builds carry actual DOM (headings, tables, carousel, banner, ad slot)
+rather than flat colour blocks.
+
+---
+
+## Prerequisites
+
+- **Node ≥ 18.19** and npm.
+- A **BrowserStack username + access key** for the account whose Percy org you want to
+  seed (this is the Basic-auth credential Percy's API uses to create projects — there is
+  no bearer "user token" for project creation).
+- A **non-production** target Percy environment (`local` / `staging` / `canary`). The CLI
+  **refuses production by design** — to seed a prod project, use the manual recipe below.
+
+```bash
+npm install
 ```
-seed-testbed --profile <local|staging|canary> [--only <feature|tier>]
+
+---
+
+## Run it (the CLI)
+
+```bash
+# 1. Make a profile for your target env (copy the example, keep secrets out of it)
+cp profiles/canary.example.js profiles/canary.js     # edit baseUrl if needed
+
+# 2. Export the credentials the profile references (never commit these)
+export BROWSERSTACK_USERNAME=<your-bs-username>
+export BROWSERSTACK_ACCESS_KEY=<your-bs-access-key>
+
+# 3. Run — full pass: a project per account tier, each with the build-type spread
+npm run cli -- --profile canary
+#   or, after `npm run build`:  seed-testbed --profile canary
 ```
 
-- Full pass (default): creates orgs/projects and builds across the account matrix
-  (free / paid / ent_global / ent_team / ai_off) for Web + App Percy, covering core
-  review states, visual-git / A-B, recurring diff, AI, approval automation, and regions.
-- `--only <feature|tier>`: a **coarse** filter for cheap retry/debug — one feature or
-  one tier. (It is *not* a per-scenario catalog.)
+On completion it prints a **run summary**: each build labeled with the feature it
+demonstrates + a dashboard deep-link.
 
-**Design & plan:** `../docs/plans/2026-07-27-001-feat-percy-feature-testbed-plan.md`
-(origin brainstorm: `../docs/brainstorms/2026-07-27-percy-feature-testbed-requirements.md`).
+### Scope it down with `--only` / `--tier`
+
+```bash
+seed-testbed --profile canary --only paid                  # all features, just the "paid" tier
+seed-testbed --profile canary --only recurring-diff        # one feature, across all tiers
+seed-testbed --profile canary --only visual-git --tier paid # one feature, one project
+```
 
-> **Status: scaffold only (plan Unit 1).** The CLI parses and routes; the orchestrator
-> and generators are not built yet. This is **not** a thin wrapper over proven parts —
-> the capture path is an unrun upstream scaffold, auto-approve is net-new, and the
-> shared-env credential/provisioning path is unvalidated. A **Phase-0 validation**
-> (plan Unit 0) must precede the capture units, and it needs live credentials
-> (see below).
+- **Features:** `core`, `visual-git`, `recurring-diff`, `ai`, `approval`, `regions`, `app-percy`
+- **Tiers:** `free`, `paid`, `ent_global`, `ent_team`, `ai_off`
 
-## `--only` keys
+### Per-feature shortcuts
 
-| Features | Tiers |
+Each feature is a one-liner that seeds just that suite's builds in a **single project**:
+
+| Command | Seeds |
 |---|---|
-| `core`, `visual-git`, `recurring-diff`, `ai`, `approval`, `regions`, `app-percy` | `free`, `paid`, `ent_global`, `ent_team`, `ai_off` |
+| `npm run core` | new / changed / unchanged / removed |
+| `npm run ab` | **A/B** variant comparison (visual-git / target-branch) |
+| `npm run recurring-diff` | recurring diff |
+| `npm run ai` | AI review over the upstream `test_bed/ai` pages |
+| `npm run regions` | `ignore` + `layout` region rules, each with a `standard` control |
+| `npm run intelli-ignore` | IntelliIgnore + sensitivity sweep, with a `standard` control |
+| `npm run approval` | auto-finalization, supersede, auto-approve |
+| `npm run app` | App Percy *(not yet wired)* |
+| `npm run seed:all` | the full matrix (all tiers × all features) |
+
+Each defaults to `--profile canary --tier paid`; override with env vars:
+
+```bash
+PROFILE=staging TIER=free npm run ai
+```
 
-## Depends on upstreams (not vendored)
+(You still need a `profiles/<env>.js` file and the creds exported — see above.)
 
-The testbed **shells out to** its upstream repos in place — it never copies them, so
-nothing drifts:
+---
 
-- `percy-api-seed-accounts` — the account rake + `capture.js` / `app_capture.py`.
-- `BStackAutomation-vra/percy/percy_playwright` — snapshot YAML + build helpers.
+## What gets created
 
-By default these are resolved as **siblings** of this repo (the `~/Desktop/percy` hub).
-Override for non-default checkouts:
+| Build type | How it's produced |
+|---|---|
+| **baseline / unchanged / changed / new / removed** | `percy snapshot` of a generated fixture set; adding/removing a page produces NEW/REMOVED, and `changed` moves the Growth plan price ($99 -> $129) against an approved `master` baseline |
+| **visual-git / A-B** | two branches + `PERCY_TARGET_BRANCH` (variant-A baseline vs variant-B head) |
+| **recurring diff** | two consecutive changed builds vs the master baseline |
+| **approval** | auto-finalization, supersede (same branch + `skipCache`), and auto-approve (sets a branch rule via the project API) |
+| **ai** | renders the upstream `percy_playwright/test_bed/ai` pages in place (8 baseline/changed pairs + reduce-diff) — real content for the classifier to work on |
+| **regions / intelli-ignore** | region rules (`standard` / `layout` / `ignore` / `intelliignore`) attached per snapshot via a generated Percy config; each rule build is paired with a `standard` **control** over the same fixture pair |
 
-- `PERCY_TESTBED_SEED_ACCOUNTS_DIR`
-- `PERCY_TESTBED_PERCY_PLAYWRIGHT_DIR`
+Every capture always sets `PERCY_CLIENT_API_URL` to the target env — otherwise the Percy
+CLI defaults to `api.percy.io` (prod) and builds land there silently.
 
-See `src/upstream.ts`.
+---
 
-## Profiles & secrets
+## How it works
 
-Each environment is a pluggable profile under `profiles/`. Copy an `*.example.js`
-template to `profiles/<env>.js` and fill in **secret references only** (the names of
-env vars that carry each secret) — never inline tokens/passwords/cookies. Real
-`profiles/*.js` files are gitignored; only the `*.example.js` templates are tracked.
+```
+seed-testbed --profile <env>
+  └─ for each account tier:
+       create project   → POST /api/v1/projects            (Basic auth = BS user:key)
+       fetch tokens      → GET  /api/v1/projects/<slug>/tokens   (write_only + read_only)
+       for each feature:
+         render fixture pages → `npx @percy/cli snapshot <dir> --config <cfg>`
+         poll the build to finished (read token), approve the baseline where needed
+  └─ print a feature-labeled run summary
+```
 
-- Runs only against a **non-prod allow-list** (`local` / `staging` / `canary`);
-  preprod is gated and prod is never a valid target.
-- The loader always injects `PERCY_CLIENT_API_URL = <baseUrl>/api/v1` on captures
-  (unset ⇒ builds silently land on prod).
-- Project mutations, baseline approval, and region/AI ops require a **user-level
-  principal**, not a project token; polling uses a read token. App Percy needs
-  valid **prod-hub** BrowserStack creds + a pre-uploaded `BS_APP_ID`.
+---
 
-## Credentials needed before capture (Phase-0 onward)
+## Running against production (manual)
 
-The scaffold needs nothing. Unit 0 (validation) and every capture unit need Percy +
-BrowserStack tokens configured in your shell (`/stack:percy-token-setup`, then restart
-the shell so the exports load), plus access to the target env's renderer.
+The CLI won't target prod. If you deliberately want to seed a **prod** project (it's real
+customer-facing data — create a clearly-named project and archive it after), run the same
+primitives directly:
 
-## Development
+```bash
+U=<bs-username>; K=<bs-access-key>
 
+# 1) create a project (org inferred from the creds)
+FS=$(curl -sS -X POST https://percy.io/api/v1/projects -u "$U:$K" \
+      -H 'Content-Type: application/vnd.api+json' \
+      -d '{"data":{"type":"projects","attributes":{"name":"my-testbed","type":"web"}}}' \
+    | python3 -c "import sys,json;print(json.load(sys.stdin)['data']['attributes']['full-slug'])")
+
+# 2) fetch the write token
+WT=$(curl -sS "https://percy.io/api/v1/projects/$FS/tokens" -u "$U:$K" \
+    | python3 -c "import sys,json;print(next(t['attributes']['token'] for t in json.load(sys.stdin)['data'] if t['attributes']['role']=='write_only'))")
+
+# 3) upload images in ./snapshots as a build (add PERCY_TARGET_BRANCH=master for a diff)
+PERCY_TOKEN="$WT" PERCY_CLIENT_API_URL=https://percy.io/api/v1 PERCY_BRANCH=master \
+  npx -y @percy/cli upload ./snapshots
 ```
-npm install
-npm test         # node --test via tsx
-npm run typecheck
-npm run cli -- --profile canary --only regions   # prints what it *would* run
-npm run build    # emit dist/ (bin: seed-testbed)
+
+---
+
+## Profiles & secrets
+
+- Each env is a profile under `profiles/`. Copy an `*.example.js` to `profiles/<env>.js`;
+  fill in **secret references** (env-var names) only — never inline tokens/keys. Real
+  `profiles/*.js` are gitignored; only `*.example.js` templates are tracked.
+- The loader always derives `PERCY_CLIENT_API_URL = <baseUrl>/api/v1` and enforces the
+  **non-prod allow-list** (prod refused; preprod gated).
+
+---
+
+## Limitations (known)
+
+- **Region-rule outcomes are not yet verified.** The fixtures render, the config is
+  accepted, and `algorithm` demonstrably changes results (the same layout-shift pair gives
+  0 diffs under `standard` and 4 under `layout`). But an `ignore` region on the carousel did
+  **not** reduce the build's diff count, and IntelliIgnore did not measurably suppress the
+  noise zones. `total-comparisons-diff` is per comparison, not per region, so it cannot tell
+  you whether a rule fired — verifying these properly needs per-region comparison data.
+  Treat the current expectations in the run summary as *intended*, not *confirmed*.
+- **AI** builds now use real pages, but AI output quality is not asserted (and the
+  requirements doc exempts AI from determinism).
+- **App Percy (mobile)** isn't wired to a working capture yet (needs BrowserStack app +
+  a pre-uploaded `BS_APP_ID`).
+- **Fixtures are a fixed 1280px layout**, so snapshots are pinned to that width. At Percy's
+  default 375px the price table falls outside the render and a real change produces no diff.
+- **`npm run build` output is not runnable** — `tsc` emits extensionless ESM imports, so
+  `dist/main.js` (the declared `bin`) fails to resolve. Everything runs through `tsx`.
+- **Delete:** the public API has no project delete — use `PATCH is-enabled=false` to archive
+  via API, or the Project Settings UI for a permanent delete.
+
+---
+
+## Development
+
+```bash
+npm test            # 80 unit tests (mocked HTTP + shell — nothing hits the network)
+npm run typecheck   # tsc --noEmit
+npm run build       # emit dist/ (bin: seed-testbed)
 ```
 
-Requires Node ≥ 20.
+Design docs (in the percy hub): `docs/brainstorms/2026-07-27-percy-feature-testbed-requirements.md`
+and `docs/plans/2026-07-27-001-feat-percy-feature-testbed-plan.md`.