eval(workbench): web-design-guidelines eval suite + proposed upstream changes

examples/workbench/web-design-guidelines/README.md

@@ -0,0 +1,159 @@
+# web-design-guidelines eval
+
+Eval suite for [`vercel-labs/agent-skills/web-design-guidelines`](https://github.com/vercel-labs/agent-skills) — a skill that reviews UI files for Vercel Web Interface Guidelines compliance.
+
+## Cases
+
+Nine cases, ~5 seeded violations each, one TSX sample per case. Each case targets a focused rule family so the agent isn't overwhelmed and we can isolate which families it handles well vs. poorly.
+
+### `review-product-card` — Accessibility + Focus States
+
+Sample: `workspace/ProductCard.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 15 | `<img>` without `alt` | Images need `alt` (or `alt=""` if decorative) |
+| 18 | `<div onClick>` for action | `<button>` for actions, not `<div onClick>` |
+| 21–23 | Icon-only `<button>` without `aria-label` | Icon-only buttons need `aria-label` |
+| 24–29 | `<input>` without `<label>` or `aria-label` | Form controls need `<label>` or `aria-label` |
+| 30–32 | `outline-none` className without focus replacement | Never `outline-none` without focus replacement |
+
+### `review-checkout-form` — Forms
+
+Sample: `workspace/CheckoutForm.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 17 | `<label>` without `htmlFor` | Labels clickable (`htmlFor` or wrapping control) |
+| 18–25 | `<input>` for email uses `type="text"` | Use correct `type` (`email`, `tel`, `url`, `number`) |
+| 18–25 | `<input>` missing `autoComplete` | Inputs need `autocomplete` and meaningful `name` |
+| 24 | `onPaste={(e) => e.preventDefault()}` | Never block paste |
+| 30 | Submit button `disabled` before request starts | Submit stays enabled until request starts |
+
+### `review-loading-screen` — Typography + Content Handling
+
+Sample: `workspace/LoadingScreen.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 12 | `"Loading..."` (three dots, not `…`) | `…` not `...`; loading states end with `…` |
+| 13 | Straight quotes `"..."` | Curly quotes `"..."` not straight |
+| 14 | `{fileSize} MB` without `&nbsp;` | Non-breaking spaces between number and unit |
+| 15–18 | Flex children without `min-w-0` for `truncate` | Flex children need `min-w-0` |
+| 19–23 | `recentFiles.map(...)` no empty-state branch | Handle empty states |
+
+### `review-hero-section` — Animation + Images + Performance
+
+Sample: `workspace/HeroSection.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 6 | Above-fold `<img>` missing `width`/`height` | `<img>` needs explicit `width` and `height` (CLS) |
+| 6 | Above-fold `<img>` missing `priority`/`fetchpriority="high"` | Above-fold critical images need priority hint |
+| 7–10 | `transition: 'all'` | Never `transition: all` — list properties explicitly |
+| 15–18 | Animation without `prefers-reduced-motion` consideration | Honor `prefers-reduced-motion` |
+| 23 | Below-fold `<img>` missing `loading="lazy"` | Below-fold images need `loading="lazy"` |
+
+### `review-data-table` — Performance + Typography + Content & Copy
+
+Sample: `workspace/DataTable.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 15–17 | `getBoundingClientRect` in render | No layout reads in render |
+| 32–38 | Large list `.map()` without virtualization | Large lists (>50 items): virtualize |
+| 27–36 | Numeric columns without `tabular-nums` | `font-variant-numeric: tabular-nums` for number columns |
+| 21 | `<h2>` without `text-balance`/`text-pretty` | Use `text-wrap: balance` or `text-pretty` on headings |
+| 22 | "eight projects" spelled out | Numerals for counts ("8 projects" not "eight projects") |
+
+### `review-confirm-dialog` — Touch & Interaction + Safe Areas + Hover States
+
+Sample: `workspace/ConfirmDialog.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 11–35 | Modal without `overscroll-behavior: contain` | Prevent scroll bleed to page behind |
+| 11–35 | Modal without `touch-action: manipulation` | Prevent double-tap zoom delay |
+| 11–35 | No `env(safe-area-inset-*)` for notched devices | Full-bleed layouts need safe-area-inset |
+| 26 | `autoFocus` on a non-primary confirmation input | `autoFocus` sparingly — desktop, single primary input |
+| 31–32 | Action buttons without `hover:` state | Buttons need hover state |
+
+### `review-search-page` — Navigation & State + Locale & i18n
+
+Sample: `workspace/SearchPage.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 11–12 | Filter/page state in `useState` only, no URL sync | URL reflects state |
+| 31 | Hardcoded currency: `${r.price.toFixed(2)}` | Use `Intl.NumberFormat` for currency |
+| 32 | Hardcoded date: `r.publishedAt.toDateString()` | Use `Intl.DateTimeFormat` |
+| 20 | "Acme Cloud" brand without `translate="no"` | Brand names need `translate="no"` |
+| 33 | `Delete` button with no confirmation | Destructive actions need confirmation modal or undo window |
+
+### `review-theme-toggle` — Hydration Safety + Dark Mode + Hover
+
+Sample: `workspace/ThemeToggle.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 7 | `localStorage.getItem` directly in render body | Hydration mismatch risk |
+| 25 | `<input value={accentColor}>` no `onChange` | Controlled inputs need `onChange` |
+| 14–22 | Native `<select>` no explicit `background-color`/`color` | Windows dark mode requires both |
+| 27 | `<button>` without `hover:` state | Buttons need hover state |
+| 14–27 | No `focus-visible:ring-*` anywhere | Use `:focus-visible` over `:focus` |
+
+### `review-blog-post` — Heading hierarchy + Aria + Focus + Content & Copy
+
+Sample: `workspace/BlogPost.tsx`
+
+| Line | Violation | Rule |
+|---|---|---|
+| 11–13 | `<h1>` → `<h3>` (skips `<h2>`) | Headings hierarchical |
+| 17–19 | Decorative `<svg>` without `aria-hidden="true"` | Decorative icons need `aria-hidden` |
+| 25–28 | Toast div without `aria-live="polite"` | Async updates need `aria-live` |
+| 23 | Generic button label "Continue" | Specific button labels |
+| 23 | `focus:ring-2` instead of `focus-visible:ring-2` | Use `:focus-visible` over `:focus` |
+
+## Vendored snapshot
+
+The skill normally `WebFetch`es its rules from `vercel-labs/web-interface-guidelines`. For deterministic eval, we vendor a snapshot at `references/web-design-guidelines/command.md` and tweak `SKILL.md` to read the local copy. The diff vs upstream is one section (`Guidelines Source` → local file).
+
+## Run
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...
+npx tsx ../../../src/cli.ts run-suite ./suite.yml --trials 3
+```
+
+## Models
+
+The suite runs a 3-provider mid-tier matrix:
+
+- `openrouter/anthropic/claude-sonnet-4.6`
+- `openrouter/openai/gpt-5-mini`
+- `openrouter/google/gemini-2.5-pro`
+
+## Latest results (9 cases × 3 models × 3 trials = 81 trials)
+
+| Metric | Value |
+|---|---|
+| Strict pass rate (all 5/5 per case) | 42/81 (52%) |
+| **Rule coverage rate** (violations identified / seeded) | **334/405 (82%)** |
+| Run cost | ~$5 |
+
+Per-case rule coverage: product-card 100%, checkout-form 98%, hero-section 100%, blog-post 87%, loading-screen 82%, data-table 80%, theme-toggle 69%, search-page 64%, confirm-dialog 62%.
+
+Strict pass dropped vs. the original 4-case suite because the 5 new cases cover **harder absence-type rules** (touch-action, safe-area-inset, brand `translate="no"`, etc.) that even the updated skill doesn't always catch. Rule-coverage 82% is the load-bearing metric.
+
+## Coverage of upstream `command.md`
+
+| Status | Rules |
+|---|---|
+| ✅ Graded across 9 cases | ~45 of 81 |
+| ⚠️ Skip (subjective / framework-bound / overlap) | ~36 of 81 |
+
+See [`proposed-upstream-changes/`](proposed-upstream-changes/) for the team's review of the SKILL.md + command.md changes we'd PR back.
+
+## Graders
+
+One grader per case under `checks/`, all sharing `_grader-utils.mjs`. Each reads `/work/findings.txt`, parses every `<file>.tsx:<line>` reference, and confirms each expected violation appears as one finding line that mentions a line in the accepted range AND matches a distinguishing keyword. `pass` requires all expected violations; `score` is the fraction found.

examples/workbench/web-design-guidelines/checks/_grader-utils.mjs

@@ -0,0 +1,70 @@
+// Shared grader logic for web-design-guidelines eval cases.
+//
+// Each finding is assumed to be one line in findings.txt that references
+// "<File>.tsx:<line>" (line numbers come from the agent — they're often
+// off by ±1-2 due to LLM line-counting). A violation is considered "found"
+// when at least one finding line:
+//   (a) references a line number within the violation's accepted range, AND
+//   (b) contains at least one of the violation's distinguishing keywords.
+//
+// This per-finding-line check prevents spurious cross-matches (e.g. the
+// keyword "label" from a different finding being credited to a paste rule).
+
+import { existsSync, readFileSync } from 'node:fs';
+
+export function gradeFindings({ findingsPath, file, expected }) {
+  const failures = [];
+  const found = new Set();
+
+  if (!existsSync(findingsPath)) {
+    failures.push('findings.txt was not created');
+    return emitResult({ found, expected, failures });
+  }
+
+  const text = readFileSync(findingsPath, 'utf-8');
+  const refRe = new RegExp(`${escapeRe(file)}\\s*[:#]\\s*(\\d+)`, 'i');
+  const findingLines = text.split(/\r?\n/).filter((ln) => refRe.test(ln));
+
+  for (const v of expected) {
+    for (const line of findingLines) {
+      const m = line.match(refRe);
+      if (!m) continue;
+      const lineNum = Number(m[1]);
+      if (!v.lines.includes(lineNum)) continue;
+      if (!v.keywords.some((re) => re.test(line))) continue;
+      found.add(v.id);
+      break;
+    }
+  }
+
+  return emitResult({ found, expected, failures });
+}
+
+function emitResult({ found, expected, failures }) {
+  const missing = expected.filter((v) => !found.has(v.id)).map((v) => v.id);
+  const score = found.size / expected.length;
+  const pass = found.size === expected.length;
+
+  console.log(JSON.stringify({
+    pass,
+    score,
+    evidence: [
+      `${found.size}/${expected.length} expected violations identified`,
+      ...[...found].map((id) => `+ ${id}`),
+      ...missing.map((id) => `- missing: ${id}`),
+      ...failures,
+    ],
+  }));
+  return pass;
+}
+
+function escapeRe(s) {
+  return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
+// Helper: build an inclusive line range [start, start+1, ..., end].
+export function range(start, end) {
+  const out = [];
+  for (let i = start; i <= end; i++) out.push(i);
+  return out;
+}

examples/workbench/web-design-guidelines/checks/grade-a11y-findings.mjs

@@ -0,0 +1,37 @@
+import { join } from 'node:path';
+import { gradeFindings, range } from './_grader-utils.mjs';
+
+const expected = [
+  {
+    id: 'img-missing-alt',
+    lines: range(13, 17),
+    keywords: [/\balt\b/i, /\bimg\b/i],
+  },
+  {
+    id: 'div-onclick-instead-of-button',
+    lines: range(16, 22),
+    keywords: [/\bdiv\b.*\bonclick\b|\bonclick\b.*\bdiv\b/i, /<button>|\bbutton\b/i],
+  },
+  {
+    id: 'icon-only-button-no-aria-label',
+    lines: range(19, 25),
+    keywords: [/aria-label/i, /icon-only/i],
+  },
+  {
+    id: 'input-without-label',
+    lines: range(22, 31),
+    keywords: [/\blabel\b/i, /aria-label/i],
+  },
+  {
+    id: 'outline-none-no-focus',
+    lines: range(28, 34),
+    keywords: [/outline-?none/i, /\bfocus\b/i],
+  },
+];
+
+const pass = gradeFindings({
+  findingsPath: join(process.env.WORK, 'findings.txt'),
+  file: 'ProductCard.tsx',
+  expected,
+});
+process.exit(pass ? 0 : 1);

examples/workbench/web-design-guidelines/checks/grade-animation-findings.mjs

@@ -0,0 +1,37 @@
+import { join } from 'node:path';
+import { gradeFindings, range } from './_grader-utils.mjs';
+
+const expected = [
+  {
+    id: 'img-missing-width-height',
+    lines: range(4, 8),
+    keywords: [/\bwidth\b/i, /\bheight\b/i, /\bcls\b|layout\s*shift/i],
+  },
+  {
+    id: 'above-fold-img-missing-priority',
+    lines: range(4, 8),
+    keywords: [/\bpriority\b/i, /fetchpriority/i],
+  },
+  {
+    id: 'transition-all',
+    lines: range(5, 12),
+    keywords: [/transition.*all|transition:\s*['"]?all/i],
+  },
+  {
+    id: 'animation-no-prefers-reduced-motion',
+    lines: range(13, 20),
+    keywords: [/prefers-?reduced-?motion|reduce[d-]?motion/i],
+  },
+  {
+    id: 'below-fold-img-missing-lazy',
+    lines: range(21, 25),
+    keywords: [/\blazy\b|loading=['"]?lazy/i],
+  },
+];
+
+const pass = gradeFindings({
+  findingsPath: join(process.env.WORK, 'findings.txt'),
+  file: 'HeroSection.tsx',
+  expected,
+});
+process.exit(pass ? 0 : 1);

examples/workbench/web-design-guidelines/checks/grade-blog-post-findings.mjs

@@ -0,0 +1,37 @@
+import { join } from 'node:path';
+import { gradeFindings, range } from './_grader-utils.mjs';
+
+const expected = [
+  {
+    id: 'heading-skip-level',
+    lines: range(11, 15),
+    keywords: [/heading\s*hierarch|skip[-\s]?level|h1.*h3|h3.*h1|missing\s*h2|level/i],
+  },
+  {
+    id: 'decorative-icon-no-aria-hidden',
+    lines: range(15, 21),
+    keywords: [/aria-hidden|decorative\s*(icon|svg)/i],
+  },
+  {
+    id: 'toast-no-aria-live',
+    lines: range(24, 30),
+    keywords: [/aria-live|role=['"]?status|live\s*region/i],
+  },
+  {
+    id: 'generic-button-label',
+    lines: range(21, 25),
+    keywords: [/specific\s*(button\s*)?label|generic\s*(button\s*)?label|"continue"|\bcontinue\b\s*(label|button)/i],
+  },
+  {
+    id: 'focus-not-focus-visible',
+    lines: range(21, 25),
+    keywords: [/focus-visible/i],
+  },
+];
+
+const pass = gradeFindings({
+  findingsPath: join(process.env.WORK, 'findings.txt'),
+  file: 'BlogPost.tsx',
+  expected,
+});
+process.exit(pass ? 0 : 1);

examples/workbench/web-design-guidelines/checks/grade-confirm-dialog-findings.mjs

@@ -0,0 +1,37 @@
+import { join } from 'node:path';
+import { gradeFindings, range } from './_grader-utils.mjs';
+
+const expected = [
+  {
+    id: 'modal-no-overscroll-behavior',
+    lines: range(9, 35),
+    keywords: [/overscroll/i],
+  },
+  {
+    id: 'missing-touch-action',
+    lines: range(9, 35),
+    keywords: [/touch-action/i],
+  },
+  {
+    id: 'missing-safe-area-inset',
+    lines: range(9, 35),
+    keywords: [/safe-area|env\(safe-area/i],
+  },
+  {
+    id: 'autofocus-on-non-primary',
+    lines: range(22, 32),
+    keywords: [/autofocus|autoFocus/i],
+  },
+  {
+    id: 'button-no-hover-state',
+    lines: range(26, 34),
+    keywords: [/hover:|\bhover\b\s*state|hover\s*feedback/i],
+  },
+];
+
+const pass = gradeFindings({
+  findingsPath: join(process.env.WORK, 'findings.txt'),
+  file: 'ConfirmDialog.tsx',
+  expected,
+});
+process.exit(pass ? 0 : 1);