docs(pilot-runs): publish batch-1 + batch-2 pilot summaries

docs/pilot-runs/2026-05-08-auto-improve-pilot-summary.md

@@ -0,0 +1,109 @@
+# Auto-improve-skill pilot summary — 2026-05-08
+
+## Setup
+
+Built a `tools/auto-improve-skill.mjs` wrapper + `tools/auto-improve-skill-prompt.md` template.
+Operator says "optimize `<slug>`"; orchestrator runs the wrapper via `Bash run_in_background`,
+the inner `claude -p` agent does the entire find → eval → diagnose → improve → package loop,
+writes `examples/workbench/<skill-id>/analysis.md`, exits.
+
+Branch: `feat/auto-improve-skill` (wrapper + prompt). Per-pilot output on `eval/auto-pilot/<skill-id>`.
+
+## Three pilot runs
+
+Run sequentially-ish: pilot #1 in main worktree, pilots #2 and #3 in parallel via `git worktree`
+in separate working folders. Three providers × three trials × N cases per pilot.
+
+| Skill | Classification | Status | Baseline | Final | Uplift | Iter | Plan-cost | OpenRouter |
+|---|---|---|---|---|---|---|---|---|
+| `vercel-labs/agent-browser/agent-browser` | tool-use | success | 0.56 | 1.00 | +0.44 | 1 | $3.15 | ~$2.80 |
+| `supabase/agent-skills/supabase-postgres-best-practices` | code-reviewer | success | 0.54 | 0.86 | +0.32 | 1 | $0 | ~$2.40 |
+| `anthropics/skills/pdf` | document-producer | success | 1.00 | 1.00 | +0 | 0 | $0 | ~$1.40 |
+
+3/3 succeeded. Each surfaced a distinct success path:
+
+- **agent-browser**: auto-pilot diagnosed that its own grader was over-specified (required `snapshot` for non-interactive ops, but the skill says CSS selectors are valid). Demoted the grader, +0.44 uplift mostly from grader correction. Also proposed a small additive "Quick task reference" section to upstream SKILL.md.
+- **supabase**: 9 SQL violations seeded (FK indexes, RLS, covering indexes, etc.). Auto-pilot first self-corrected its grader (line tolerance ±3 → ±8, added keyword variants), then independently rediscovered the same **two-pass workflow** pattern we found manually for web-design-guidelines (pass 1 = visible token misuse, pass 2 = absence checks). Real upstream proposal generated.
+- **pdf**: baseline already 1.00, auto-pilot triggered the "≥0.95 → exit clean, no proposal" path correctly. Did NOT manufacture problems. Noticed and noted that upstream's REFERENCE.md / FORMS.md links are 404.
+
+## Costs
+
+- OpenRouter (matrix runs): ~$6.60 total across 3 pilots.
+- Plan budget (the inner `claude -p` self-reported `total_cost_usd`): only #1 hit the cap.
+  Pilot #1 first attempt blocked at $3.42 from the docker-permissions issue. Pilot #1c with
+  `--budget 15` settled at $3.15. Pilots #2 and #3 reported $0 (likely under tracking floor
+  or didn't iterate enough to register).
+- Wall clock: ~50 min for 3 parallel pilots (vs ~150 min sequential).
+
+## Auto-pilot capabilities validated
+
+1. **Correct skill-shape classification** in all 3 cases (`tool-use`, `code-reviewer`, `document-producer`).
+2. **Self-correction of own grader bugs** before diagnosing the underlying skill — happened in 2 of 3 pilots without operator nudging. Same patterns we manually applied (line-tolerance widening, hyphenated regex variants, keyword alternations).
+3. **Pattern transfer**: the auto-pilot rediscovered the "two-pass workflow for absence-type rules" insight on supabase — a different skill in a different rule space — confirming the pattern generalizes.
+4. **Clean exit on already-good skills**: pdf ran 36/36 trials passing at baseline; auto-pilot did not manufacture changes.
+5. **Distinguishing skill problem from grader problem**: agent-browser caught grader-over-specification, separated it from skill quality.
+
+## Issues found in v1 of the auto-pilot
+
+1. **"Always: commit" step unreliable.** Pilots #1b and #2 didn't reach it — case files were left untracked in the worktree. Fix: hoist the commit step earlier (right after analysis.md is written), or split the prompt into two `claude -p` invocations (build + analyze).
+2. **`--max-budget-usd 3.50` is too tight** for runs that need any real iteration. Pilot #1's first real-data attempt hit the cap mid-modification. Bumping to $15 worked. Sensible default for v2: $7-10.
+3. **Phase 4 grader-fix iteration eats one of the two iteration slots.** The agent often spends iteration 1 fixing graders and only has one shot at modifying the skill. Fix: pre-bake known grader-tuning patterns into `_grader-utils.mjs` so the agent doesn't have to discover them, or count grader-only fixes separately from skill-modification iterations.
+
+## Patterns we should bake into v2
+
+From pilots and prior manual runs, these recurring techniques are stable enough to embed as defaults:
+
+**Optimizing patterns** (bake into prompt as Phase-4 priors):
+
+- Two-pass workflow (pass 1 visible / pass 2 absence) for code-reviewer skills
+- Per-element checklists for skills with rule-by-element structure
+- BAD/GOOD examples for anti-pattern and absence-type rules
+- "Verify-tool-installed" nudge for tool-use skills (agents fall back to `curl`/`npm i`)
+
+**Grader-reliability patterns** (bake into `_grader-utils.mjs`):
+
+- Default `±5–8` line tolerance
+- Hyphen-tolerant regex (`/empty[-\s]+state/`)
+- Per-finding-line keyword matching
+- Multiple keyword variants (`/cover/i` for both "covering" and "does not cover")
+
+**Default seeded violation types** (bake into Phase-2 instructions):
+
+- For code-reviewer: ≥1 visible-token, ≥1 missing-attribute, ≥1 missing-branch, ≥1 anti-pattern, ≥1 state-machine
+- For tool-use: ≥1 reaches-for-fallback, ≥1 wrong-flag, ≥1 missing-step
+- For document-producer: ≥1 missing-field, ≥1 wrong-format, ≥1 edge-case-input
+
+## Decision points for the team
+
+1. **Continue scaling.** With these results, "optimize 10 skills" is a sequential loop the
+   orchestrator already supports (just call the wrapper N times). With worktrees, N=3 in
+   parallel is also straightforward. Cost per skill ~$2-3 OpenRouter + plan-tokens.
+
+2. **Tighten the prompt before scaling.** The "Always: commit" issue and the budget-too-tight
+   issue are real and would cost a fraction of one pilot to fix. ~30 min of work for v2.
+
+3. **Build the lessons doc.** A `tools/auto-improve-skill-lessons.md` referenced by the
+   prompt as Phase-4 prior, updated after every pilot. Compounds: pilot N benefits from
+   patterns 1..N-1. Not started; sub-project for after the next batch.
+
+4. **Skill-batch parallelism.** Worktree-per-pilot worked. For 10 skills, 3-way parallel
+   would land in ~3-4 batches (~3 hours). 5-way is also feasible if the dev machine has
+   the resources.
+
+## Reproducing the pilots
+
+```bash
+cd /home/yuqing/Documents/Code/skill-optimizer
+git checkout feat/auto-improve-skill
+node tools/auto-improve-skill.mjs <owner>/<repo>/<skill-id> [--budget 15]
+
+# Output: examples/workbench/<skill-id>/{analysis.md, suite.yml, ...}
+# Branch: eval/auto-pilot/<skill-id>
+```
+
+For parallel runs, use git worktrees:
+
+```bash
+git worktree add ../wt-pilot-2 -b auto-pilot/wt-2 feat/auto-improve-skill
+cd ../wt-pilot-2 && node tools/auto-improve-skill.mjs <slug-2> --budget 15
+```

docs/pilot-runs/2026-05-09-auto-improve-batch-2-summary.md

@@ -0,0 +1,100 @@
+# Auto-improve-skill batch 2 summary — 10 pilots, 8 success, 0 failures
+
+## Setup
+
+- **Wrapper version:** v1.1 + #3 (atomic write-and-commit, $10 default budget, lessons.md, pre-baked grader helpers)
+- **Skills:** ranks 5–14 from the prioritized top-N list (skips the 4 already covered in batch 1: web-design-guidelines, agent-browser, supabase, pdf)
+- **Parallelism:** 10 git worktrees, hardlinked `node_modules`, fired simultaneously
+- **Wall clock:** ~50 min (slowest pilot to longest), down from estimated ~150 min sequential
+
+## Headline results
+
+| # | Skill | Classification | Status | Coverage | Mods | Notes |
+|---|---|---|---|---|---|---|
+| 1 | `anthropics/skills/pptx` | document-producer | ✅ success | 0.85 → 0.85 | 0 | grader cal raised raw 0.74 → 0.85; gpt-4o-mini fails entirely (model gap) |
+| 2 | `vercel-labs/next-skills/next-best-practices` | code-reviewer | ✅ success | 0.80 → 0.975 | 0 | grader cal only — skill already strong |
+| 3 | `firebase/agent-skills/firebase-auth-basics` | code-reviewer | ✅ success | 1.00 → 1.00 | 0 | reclassified from prior `tool-use` |
+| 4 | `firebase/agent-skills/firebase-hosting-basics` | code-patterns | ✅ success | 0.89 → 1.00 | 1 | Recipe A + E added a Configuration Review section |
+| 5 | `expo/skills/building-native-ui` | code-patterns | ✅ success | 0.99 → 0.99 | 0 | 17/18 trials — single gpt-5-mini miss accepted as noise |
+| 6 | `google-labs-code/stitch-skills/shadcn-ui` | code-patterns | ✅ success | 0.82 → 0.89 | 1 | Recipe A + D — Gemini's wrong-location miss rate dropped 100% → 0% |
+| 7 | `expo/skills/native-data-fetching` | code-reviewer | ✅ success | 1.00 → 1.00 | 0 | already-good |
+| 8 | `firecrawl/skills/firecrawl-build-scrape` | code-patterns | ⚠️ uplift-too-small | 0.84 → 0.89 | 2 | +0.05, exactly on threshold; gpt-4o-mini verbosity floor caps it |
+| 9 | `vercel-labs/next-skills/next-upgrade` | code-reviewer | ⚠️ uplift-too-small | **0.83 → 0.76** | 2 | **regression** — modifications hurt; new failure mode surfaced |
+| 10 | `github/awesome-copilot/prd` | document-producer | ✅ success | 1.00 → 1.00 | 0 | sonnet API errors, judged on 12 valid trials from gpt-5-mini + gemini |
+
+**8/10 success • 2/10 uplift-too-small • 0/10 blocked or budget-exceeded**
+
+## Cost
+
+- OpenRouter spend during batch: **~$21.30** ($40.65 used – $19.35 prior to batch start)
+- Per-pilot avg: **$2.13** (well under the $3.50 budgeted)
+- Plan-token spend (inner `claude -p`): each pilot reported between $0.00 and $1.00 — no pilot hit the $10 wrapper cap
+
+## What v1.1 + #3 actually delivered
+
+The pilots demonstrate the prompt improvements working as intended:
+
+1. **"Atomic write-analysis-and-commit" worked.** **All 10 inner agents committed cleanly.** No manual recovery needed (vs batch 1 where 2 of 3 needed manual commits).
+2. **Recipe citations by letter.** Pilots 4, 6, 8 explicitly cited Recipe A / D / E from `lessons.md` in their analysis bullets. They didn't rediscover the patterns from scratch.
+3. **"Grader-vs-skill check first" worked.** Pilots 1, 2, 4, 6, 8, 9 all did iteration 0 grader calibration before counting against their iteration budget. Saved meaningful budget on pilots 2, 4, 6.
+4. **`looseRange` / `tolerantKeyword` pre-baked helpers** — used in graders the auto-pilot wrote without rediscovering the patterns. Several pilots had to widen specifically for gpt-4o-mini drift (range 8 → 12 or 16) which is new signal worth adding to lessons.md.
+5. **"Don't manufacture problems"** worked in all 5 already-good cases (3, 5, 7, 10, plus pilot 1 after grader cal). None proposed unnecessary changes.
+
+## New patterns surfaced — worth adding to `lessons.md`
+
+### Optimization patterns
+
+- **(NEW) Recipe F? — Don't add bash commands for small models.** Pilot 9 added bash grep commands to `next-upgrade`'s SKILL.md. gpt-4o-mini tried to *execute* them rather than reading files, dropping coverage from 0.83 to 0.69. **Anti-pattern.** When skill is aimed at small/cheap models, prefer pure declarative wording over executable commands.
+
+### Failure modes
+
+- **CLI fabrication on "upgrade-style" skills.** gpt-4o-mini will hallucinate a `npx <something>-upgrade` CLI for any skill whose name suggests transformation/upgrade work, then write the error message as findings. Distinct from the agent-browser `curl` fallback (where the CLI exists but the model picks the wrong tool). Worth its own anti-pattern entry.
+- **Verbosity floor on gpt-4o-mini.** Confirmed across pilots 8, 9 — emits 3-4 line responses, sometimes drops trailing rules entirely. Rules requiring multi-finding output above this floor are systematically under-detected.
+
+### Grader patterns
+
+- **(NEW) Per-model line tolerance.** sonnet/gemini drift 0–3 lines; gpt-4o-mini drifts 6–15 lines. The `looseRange` default of ±8 is calibrated for the first two but undertuned for the third. Future graders should default to `looseRange(N, 12)` or use per-model tolerance maps.
+
+### Skill-shape edge cases
+
+- **Repo path conventions vary.** `expo/skills` uses `plugins/expo/skills/<id>/SKILL.md` (not the canonical `skills/<id>/SKILL.md`). Pilots 5 and 7 both surfaced this and adapted. Worth noting in Phase-1 instructions.
+
+## Branches pushed
+
+- `eval/auto-pilot/batch-2-2026-05-09` (consolidated, all 10 cherry-picked)
+- 10 individual `eval/auto-pilot/<skill-id>` branches (for per-skill review)
+
+## What to PR upstream
+
+Three pilots produced real, additive proposals:
+
+| Skill | Uplift | Where the change goes |
+|---|---|---|
+| firebase-hosting-basics | 0.89 → 1.00 | `firebase/agent-skills` |
+| shadcn-ui | 0.82 → 0.89 | `google-labs-code/stitch-skills` |
+| firecrawl-build-scrape | 0.84 → 0.89 | `firecrawl/skills` |
+
+**Skip from PR queue:**
+
+- All 5 baseline-already-good skills (no changes warranted)
+- pilot 9 (next-upgrade) — modifications regressed; needs human review or a different approach (probably "drop bash commands, use BAD/GOOD only")
+- pilot 8 (firecrawl-build-scrape) is on the bubble at +0.05 — judgment call
+
+## Decision points for the team
+
+1. **Scale further.** With v1.1+#3 working, batch 3 of 10 skills should land in another ~50 min for ~$25 OpenRouter. Plenty of remaining slugs in the top-N (15–47).
+2. **Lessons.md v1.2 update.** Add the patterns from this batch (CLI fabrication, gpt-4o-mini line drift, repo-path variants, "don't add bash for small models"). 30 min of doc work that compounds for batch 3.
+3. **Drop gpt-4o-mini from default matrix.** Repeated capability gap (verbosity floor + CLI fabrication + line drift) is dragging multiple pilots' scores. Switching the matrix to sonnet/gemini/another-mid-tier would likely lift batch coverage by 5-10pp without any skill changes. Worth piloting.
+
+## Reproducing
+
+```bash
+# This batch can be reproduced from a fresh checkout of feat/auto-improve-skill:
+cd /home/yuqing/Documents/Code/skill-optimizer
+git checkout feat/auto-improve-skill
+node tools/auto-improve-skill.mjs <slug>
+
+# For parallel batches, use git worktrees (see batch script in this commit's Setup section)
+```
+
+Cumulative spend: $40.65 of $60 OpenRouter credits.

docs/pilot-runs/README.md

@@ -0,0 +1,39 @@
+# Auto-improve-skill pilot runs
+
+Summaries of batched runs of the `tools/auto-improve-skill.mjs` auto-pilot
+against public agent skills from our prioritized top-N list. Each summary
+documents what skills ran, what the auto-pilot proposed, what worked, what
+didn't, and what changes we should make to the prompt before the next batch.
+
+The per-skill eval artifacts (suite, graders, vendored upstream, proposed-upstream-changes/)
+live on `eval/auto-pilot/<skill-id>` branches and the consolidated
+`eval/auto-pilot/batch-<n>-<date>` branches.
+
+## Index
+
+- [`2026-05-08-auto-improve-pilot-summary.md`](./2026-05-08-auto-improve-pilot-summary.md)
+  — Batch 1, 3 skills (agent-browser, supabase-postgres-best-practices, pdf).
+  Validated end-to-end. 3/3 success.
+- [`2026-05-09-auto-improve-batch-2-summary.md`](./2026-05-09-auto-improve-batch-2-summary.md)
+  — Batch 2, 10 skills (pptx, next-best-practices, firebase-auth-basics,
+  firebase-hosting-basics, building-native-ui, shadcn-ui, native-data-fetching,
+  firecrawl-build-scrape, next-upgrade, prd). 8/10 success, 2/10 uplift-too-small.
+
+## How to run a new batch
+
+```bash
+# Single skill from the main repo:
+node tools/auto-improve-skill.mjs <owner>/<repo>/<skill-id> [--budget 10]
+
+# Parallel batch via git worktrees:
+for i in {1..N}; do
+  git worktree add ../wt-pilot-$i -b auto-pilot/wt-batch-$i feat/auto-improve-skill
+  cp -al node_modules dist ../wt-pilot-$i/
+  cp .env ../wt-pilot-$i/
+done
+
+# Then fire one wrapper invocation per worktree in parallel.
+```
+
+After all pilots complete, cherry-pick each `eval/auto-pilot/<skill-id>` onto
+a consolidated batch branch and open a PR.