feat: Percy feature testbed — one-command build seeder (Units 1–10)

[AkashBrowserStack]

What & why

One command — seed-testbed --profile <env> — populates a Percy environment with real renderer-backed builds exercising every Percy feature, for manual QA. Default is a full pass (every account tier × every feature); --only <feature|tier> narrows it for targeted re-runs.

Builds render actual DOM via percy snapshot. The earlier approach uploaded solid-colour PNGs, which proved create project → token → build → diff and nothing else: the AI classifier had nothing to classify and region rules had nothing to reason about, so both were smoke builds by construction.

Origin: brainstorm → plan (docs/plans/2026-07-27-001-feat-percy-feature-testbed-plan.md in the percy hub).

Verified live

Every build below is on prod org 9560f98d, produced by this branch's generators.

Region rules ✅ — percy-testbed-rich-rgd7733

Build	Rule	Diffs
52700854	ignore on the carousel, carousel-only change	0	✅ as expected
52700911	control — standard, same pair	4	✅ as expected
52700952	ignore on carousel, all 4 zones changed	4	expected — 3 zones still differ
52700994	layout on a displaced sidebar	4	⚠️ see below
52701059	control — standard, same pair	0	⚠️ see below

The first pair is the proof. Identical fixtures, the carousel is the only thing that changed: standard flags it, ignore takes the build to zero. Region config demonstrably reaches Percy and does what it says.

That pair exists because the earlier version of this test could not fail — it changed four zones and ignored one, leaving three still changed, so the count stayed at 4 whether or not the rule fired. An unfalsifiable test reads as "the feature is broken" when it is really saying nothing. Isolating the change fixed that.

⚠️ Layout runs opposite to expectation. On an identical pair the layout rule flags 4 and standard flags 0. The rule may be designed to detect layout change rather than suppress it — that is not confirmed, so the build asks QA to compare the pair rather than asserting a direction.

AI ✅ — all three R12 capabilities — percy-testbed-rich-bug322a

Build	Snapshots	Diffs	AI reduced
52700252 — visual bug classification	2	4	8
52700352 — AI build summary, 8 realistic page pairs	8	24	10
52700474 — reduce-diff	1	4	—

ai-details.total-diffs-reduced was null on every build before this branch's fixes; it now carries real values. The bug-classification build ships five deliberate defects — unreadable carousel text, banner copy clipped mid-word, a misaligned price row with an overlapping value, a failed ad image, and the sidebar overlapping the main column — because Percy tags such regions visual_quality: 'irregularity', which needs something that looks broken rather than merely changed.

Core review states ✅ — percy-testbed-rich-rich0180

Build	Branch	Snapshots	Diffs
52695689 baseline (approved)	master	2	16/16
52695709 changed	feature/seed-change	2	4/16

IntelliIgnore ⚠️ — runs, but not yet distinguishable — percy-testbed-rich-fix4b7b

Build	Rule	Diffs	AI reduced
52699608	intelliignore, 4 noise zones	4	—
52699656	control — standard	4	4
52699692	intelliignore, noise + price change	4	4
52699720 / 52699755	sensitivity 0 / 4	4 / 4	8 / —

AI reduction happens, but the pattern does not track the rules — the standard control also shows reduction, suggesting project-level IntelliIgnore reduces across all builds independently of the per-region algorithm. These scenarios need the same isolated-change treatment the ignore test just got before they prove anything.

Bugs this work surfaced

• Settings writes were silent no-ops. editProject sent snake_case keys; Percy answers 200 for unknown keys and ignores them. Verified against prod: ignore_carousels_enabled → stays false, ignore-carousels-enabled → true, both 200. Keys are now dash-cased, and editProject reads the response back and throws when a value did not stick. The same bug was already in setAutoApprove, so auto-approve had been quietly doing nothing — and two existing tests asserted the same wrong key the code sent, which is why it survived.
• ai-enabled defaults to false, so AI and IntelliIgnore never ran. Both generators now enable it.
• Approve returns 409 on orgs whose default branch auto-approves; reviewBuild treated that as failure, which would break every generator that approves a baseline.
• Percy's default 375px width left the fixed-width fixture's price table outside the render, so a genuine change produced no diff at all. Snapshots are pinned to 1280.

What's in this PR

• Fixtures (src/fixtures/) — a zoned storefront generated from one source so variants cannot drift, with the zones Percy's algorithmConfiguration knows about (carousel, ad, banner, dynamic timestamp) plus a price table as the signal and a sidebar for layout. Deterministic: no Date.now, no randomness, so re-runs reproduce pixels.
• Generators — new intelli-ignore, regions covering ignore/layout with controls, ai covering all three R12 capabilities (build summary and reduce-diff render the upstream percy_playwright/test_bed/ai pages in place, no vendoring; bug classification uses in-repo fixtures because upstream's point at external sites).
• GeneratedBuild.expectation — every build states what QA should see, printed in the run summary.
• Removed src/images.ts — dead with the upload path.

Tests

82 unit tests (mocked HTTP + shell — nothing hits the network), tsc --noEmit clean. npm test.

Known limitations

• IntelliIgnore scenarios are not yet decisive (above).
• Layout-rule direction unconfirmed (above).
• AI output quality is not asserted — the requirements doc exempts AI from determinism.
• App Percy isn't wired to a working capture (needs a BrowserStack app + BS_APP_ID).
• A full seed:all across tiers still assumes the seeded account matrix exists on the target env.
• npm run build output is not runnable (pre-existing): tsc emits extensionless ESM imports, so dist/main.js — the declared bin — fails to resolve. Everything runs through tsx.

Notes

• Upstreams are used by shelling out / reading in place — no vendoring, so nothing drifts.
• No secrets committed — only profiles/*.example.js templates.
• Builds live in throwaway prod projects and Percy ages them out; re-run the seeder if they expire before review.

🤖 Generated with Claude Code

[rishigupta1599]

Testbed is very basic just color change, it should have complex dom to verify, to get proper AI diffs intelli ignore diffs etc....