v3.0.0 - five phase commands, opt-in gates, and evals - #15
Merged
Merged
Conversation
Groundwork for v2.4.0, the gates release described in PRD-graph-orchestration.md. Promotes the workflow from advisory prose to enforced state transitions. - hooks/lib/state.js: front-matter parse/serialize/patch, stage resolution (none/bootstrap/feature), the stage-1 to stage-2 handoff predicate, and the ULTIMATE_WORKFLOW_GATES escape hatch. Node builtins only. - hooks/lib/gates.js: the five gate predicates as pure functions. No filesystem, environment, or stdout access, so the gate table is testable row by row and the vendor adapters stay trivial. - hooks/dev/record.js: payload recorder for the tier-2 adapter fixtures. Contract tests run against recorded payloads rather than documentation, because the point is to catch where vendors differ from their docs. - 52 unit tests, node:test only, no package.json and no dependency. - validate.yml: new test job on Node 22. The PRD and the plan are committed alongside the code: the PRD supersedes the v0 LangGraph-over-MCP design, and the plan spans two releases so it outlives the usual transient PLAN-*.md lifecycle. Force-added past .gitignore for that reason; the ignore rule stays for ordinary per-feature plans. Two gotchas recorded in CLAUDE.md: `node --test <dir>` resolves the directory as a module path and fails, and the absence of package.json is a deliberate architectural constraint rather than an oversight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recording real Claude Code payloads surfaced two defects that the tier-1 tests could not catch, because those tests only used relative paths. tool_input.file_path is always absolute. isTestPath therefore matched the whole absolute path, so a project living under any directory named "test" -- /home/k/test/myproject, say -- classified every one of its files as test code and silently disabled gate G2. The target is also routinely outside cwd: scratchpad and temp-file edits were being judged against this project's plan, so an unconfirmed plan here would have blocked edits to unrelated files elsewhere on disk. toProjectPath(cwd, target) now relativises and returns null for anything outside the project, which preEditGate treats as allow. Windows comparison is case-insensitive and POSIX case-sensitive, detected from the path shape rather than process.platform so it stays testable on either host. Eleven tests added, 52 to 63. One of them asserts that the unrelativised path does misfire, so the reason relativisation is required is covered rather than merely commented. Also: recorder takes --vendor and --out argv, since hook configs cannot reliably set environment variables across shells; .cursor/hooks.json registers it for the Cursor recording pass; fixtures/ is ignored as per-machine development output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recorded real Cursor payloads across three rounds and settled the open question from PRD section 11.1. Cursor honours a preToolUse deny for Read and ignores it for Write: the file is created, then afterFileEdit and postToolUse both fire. Confirmed on 3.17.19 across two independent runs, with the probe re-verified by replaying Cursor's own recorded payload and observing exit 2 plus a deny on stdout. Cursor therefore carries 3 of 5 gates preventively. G1 and G2 degrade to detect-and-correct: afterFileEdit records the violation, and stop returns followup_message instructing a revert, bounded by Cursor's loop_limit. The bigger finding is incidental. Cursor prefixes every hook payload with a UTF-8 BOM, undocumented, which makes JSON.parse throw. A hook written the obvious way -- parse, inspect, decide -- catches the exception, decides nothing, and exits 0. Installed, silent, enforcing nothing, and passing every test written against documented payloads. It cost two invalid rounds here because the probe itself had the bug. Both dev hooks now use an explicit stripBom() codepoint check rather than a regex containing a literal BOM, which is invisible in source and survived two rounds of direct inspection. Two gotchas recorded in CLAUDE.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The postToolUse payload following a denied Write carries success:true on both measured runs, so the denial is not surfaced to the agent at all. There is no rejected-tool-call path to fall back on: the model cannot learn from the block, and the only available signal is one the gate manufactures afterwards via afterFileEdit plus a stop follow-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recorded transcripts carry many beforeSubmitPrompt/stop pairs with status: aborted and no tool calls between them -- cancelled or superseded generations, ordinary in interactive use. Since followup_message auto-submits as a new user message, emitting it on those burns the loop_limit and pesters the user for turns that did nothing. Only visible in a real transcript containing real cancellations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the vendor adapters, the dispatcher, and a single entry point, and registers them for Claude Code and Cursor. 104 tests, including end-to-end runs through gate.js. - lib/adapters.js translates each host's wire format to one canonical shape and back. Contract tests run against payloads RECORDED from real sessions, not hand-written from docs. - lib/dispatch.js routes a normalised event to a gate. Pure, so the gate tables in PRD sections 5 and 5.1 are testable row by row. - gate.js is the only entry point: one script registered for every event, dispatching on the normalised event rather than one script per gate. - Dev recorder and deny probe unregistered; .cursor/hooks.json now carries the real gates. Three decisions that came from measured behaviour rather than documentation: On Cursor the pre-edit gate deliberately does NOT emit a deny for writes. A deny there is dropped and the following postToolUse reports success:true, so emitting one would look like enforcement while providing none. It records the violation and lets the stop follow-up instruct a revert. Verification result comes from WHICH event fires, not from a payload field: no host puts an exit code in the post-tool payload, but all except Gemini fire a distinct tool-failure event. The stop gate ignores turns whose status is not "completed". Cursor fires stop on cancelled and superseded generations, and followup_message auto-submits as a user message, so acting on those burns the loop_limit. README gains a host-support matrix stating plainly that G1 and G2 are detect-only on Cursor, and that the Codex and Gemini adapters are written to documented contracts nobody has exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…2.4.0) BREAKING: the plugin name changes from the-ultimate-workflow-guidelines to ultimate-workflow. The name namespaces every component the plugin ships, so commands and skills become ultimate-workflow:<name>. Users reinstall once; the repository, marketplace entry, and GitHub URL are unchanged. Done now because the old name already produced the-ultimate-workflow-guidelines:the-ultimate-workflow-guidelines in the skill list, and the workflows planned for v3.0.0 would have made it worse. BREAKING: the workflow gates are on by default. They are inert in any repository without a PRD.md or PLAN-*.md, ULTIMATE_WORKFLOW_GATES=off disables them, and they fail open. - State front matter added to plan-template.md and prd-template.md, with a note explaining that plan_confirmed and tests_confirmed are set only after the user has actually confirmed -- not to clear a block. - A "Gates" section mirrored into all three copies of skill 1's body (SKILL.md, CLAUDE.md, rules/*.mdc) in this commit, per mirror discipline. - validate.yml gains a manifest name-consistency assertion beside the existing version check. The marketplace's own top-level name is deliberately excluded: it identifies the marketplace, not the plugin. - README structure table covers the new hooks; install command updated. - CHANGELOG documents both breaking changes and the known limitations, including that Cursor cannot prevent a write and that the Codex and Gemini adapters are unverified. 104 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
198 lines to 121. The old one read as a reference manual: a 27-row file inventory before the reader learned what the thing does, and the workflow described only in the abstract. Now it opens with the problem, then walks one real feature -- the lifecycle gates in this repo -- through the actual five steps, including the point where recorded payloads contradicted the vendor docs and the workflow stopped rather than guessed. Concrete beats a bullet list of virtues. Cut the file inventory to a five-line tree, compressed the principles to one sentence, and trimmed the install section to the commands plus what each path omits. Kept the Cursor limitation and the docs-only caveat in full: those are correctness, not marketing, and shortening them would mislead. Also drops the emoji in the old line 5, which contradicted the no-emoji rule this plugin enforces on everyone else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dogfood cycle, built under the gates it extends. 104 to 136 tests. The problem it solves is one this session hit directly: gates were registered in settings, the code was correct, and nothing enforced anything for twenty minutes because the host reads hook config at startup. A hook that was never loaded, a hook that crashed, and a hook that correctly allowed are indistinguishable from outside -- all three are silence. - lib/heartbeat.js records every gate invocation to .ultimate-workflow/heartbeat.json: last_seen, a running count, and the 20 most recent decisions. Bounded by construction, gitignored, and every function swallows its own errors so diagnostics can never affect a decision. liveness() separates never / stale / live / disabled. - status.js reads heartbeat plus state and reports both. It asks the real gate predicates what would block rather than reimplementing them, so it cannot drift from actual behaviour. - gate.js records a heartbeat after every decision, in its own try/catch. Also fixes a real defect found while planning this: gate_violation was written once and never cleared, so after a detect-only host let an edit through, every later stop message kept quoting the resolved violation and telling the user to revert an edit that no longer existed. A green verify now clears it, and a newly-allowed pre-edit clears it immediately rather than waiting for the next green run. O3 answered: the Stop gate never got in the way. Edit source, run tests, turn ends clean -- and running tests before finishing is something you do anyway. The only turn it would have blocked was one with unverified source changes, which is the case it exists to catch. Silent non-enforcement, not strictness, is the real risk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cursor returns to the plugin as it was before the gates work: two skills and the alwaysApply rule, no hooks. The rename to ultimate-workflow is kept, since that is orthogonal to enforcement. - .cursor/hooks.json removed. - Gates section removed from rules/*.mdc, which is Cursor-only. Documenting gates there would describe something that never runs. - The host note in SKILL.md and CLAUDE.md now says plainly that gates are Claude Code only, rather than claiming reduced enforcement on Cursor. - README replaces the four-host matrix with the measured scope. A deliberate mirror-discipline exception is recorded in the README: the Gates section lives in SKILL.md and CLAUDE.md but not in rules/*.mdc. Without that note a later session would "fix" the drift and reintroduce exactly the problem this removes. The Cursor adapter stays in adapters.js with its measurements intact. It is tested, costs nothing to keep, and is the record of what was observed. It is simply not registered anywhere. Also adds PLAN-graph-native.md, which supersedes the gates-first sequencing in the PRD, and records the step 0 verification: agentType inside a workflow does honour the agent definition's tool restrictions, including against the deferred tool registry. 136 tests still green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 1 of the graph-native rebuild. Read-only by construction, so it can be proven useful before anything that writes gets built. - agents/uw-harvester.md: Read, Grep, Glob, Bash. No write tool, so the phase returns proposals and a human applies them. Doc edits are exactly what accumulates silent noise when an agent can write them unattended. - workflows/harvest.js: three readers in parallel (plan files, git history, what is already recorded), then one router. The barrier is justified: the router cannot dedupe a candidate until it has the existing inventory. Validated over two real runs against main..HEAD. The anti-quota check is the one that mattered: run 2 re-surfaced all four Gotcha candidates and discarded every one as already recorded, citing CLAUDE.md line numbers. It manufactured nothing to fill space. Three proposals applied to CLAUDE.md. One was a replacement rather than an addition: the harvester found that the day-old hook-startup Gotcha had been invalidated by a3200d7 removing .cursor/hooks.json. Nothing asked it to audit existing entries -- it did so while deduping. That is now specified in the agent definition instead of left to luck, because an always-loaded instruction that has gone wrong is worse than a missing one. Two findings recorded in the plan rather than glossed: Cost is 158k subagent tokens and about 5.5 minutes per run. This is a once-per-feature command. Two runs on near-identical input produced different proposals. The orchestration is deterministic; the agents inside it are not. memory/enforcement.md is deferred until graph-native lands, and its draft deliberately not saved -- a stale topical is worse than a missing one, and regenerating from current state is what harvest is for. 136 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The review phase found these in code that had 136 passing tests. Every one of those tests exercised a pure predicate; none touched the wire format the host validates or the defaults actually shipped. The bugs lived in the gaps. Finding 0 -- the escalation never reached anyone. respond() hardcoded hookSpecificOutput.hookEventName to UserPromptSubmit, but the G4 escalation fires on PostToolUse/PostToolUseFailure, and the host throws "Hook returned incorrect event name" and discards the whole result. The round cap counted correctly for three rounds and then said nothing. gate.js now passes the event that actually fired. Finding 1 -- front matter corrupted itself. format() escaped with JSON.stringify; coerce() stripped the quotes without unescaping. Every patchState doubled the backslashes in a Windows test_command, so isVerifyCommand could never match, dirty never cleared, and G3 blocked every turn while quoting an ever-longer mangled command into the user's own plan file. Verified compounding at 2, 4, then 8 backslashes per separator. Finding 2 -- the shipped template bricked the stop gate. With test_command: "", roundCap never runs, so neither dirty nor escalated can ever become true and every turn end is blocked until the host's consecutive-block override fires. A gate that cannot be satisfied is worse than no gate: stopGate now steps aside when nothing is configured, and status.js reports G3 as inactive rather than staying silent. Finding 6 -- shell writes left the tree looking clean. PreToolUse matched only the file-editing tools, and the harness's own auto-mode guidance pushes agents toward sed and heredocs. Tool-name matching cannot prevent that, but the tree must not be reported verified when a shell command may have written to it. Over-marking costs one test run; under-marking loses the requirement. Finding 5 -- the fixture-backed contract tests were gitignored out of CI, so the tests written to catch vendor drift were the ones not running. Fixtures are now sanitised and committed, curated to one per payload shape. Tests for a host with no recordings skip with a message naming what is missing and how to produce it, rather than hard-failing or passing silently. Sanitising the fixtures corrupted them: a doubled backslash collapsed in the tool pipeline before Node saw it, producing invalid JSON escapes. The Cursor recordings were never committed and are unrecoverable; their findings survive in the Gotchas, the PRD, and the adapter comments, but the evidence does not. Third occurrence of that escaping failure this session, now a Gotcha. 148 tests: 145 pass, 3 skip with stated reasons. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four orthogonal lenses in parallel, then an adversarial pass that tries to refute what they found. - agents/uw-reviewer.md: Read, Grep, Glob, Bash. One definition serving both roles -- lens reviewer or verifier -- because the discipline is the same: a finding needs a file, a line, and a concrete failure scenario, or it is a suspicion. The verifier defaults to refuted when uncertain, since a false finding costs more than a missed one: it burns trust in every true finding beside it. - workflows/review.js: default rigor uses one batch skeptic; --rigor high escalates to one per finding in isolated contexts, which is more rigorous and roughly triples the bill. Refuted findings are returned, not dropped. A verifier that silently filters cannot itself be audited, and a review that quietly discards has started lying by omission. Output separates confirmed, refuted, and unverified. First real run against this branch: 13 findings, 8 confirmed, 5 refuted, 0 unverified, in 5 agents. The refutations were not hand-waving -- one verifier ran a live experiment to disprove a claim about which events fire, another read the host binary to check whether a settings key existed. Cost: 460k subagent tokens over 18.6 minutes, roughly 3x /harvest. The agent count stayed in budget; the depth is what cost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 3 of the graph-native rebuild. Both return their output for review rather than writing it, which settles the invariant: only /build writes, and it is the phase gated behind two confirmations. Every other phase is read-only by construction, not by discipline. - agents/uw-explorer.md: Read, Grep, Glob. No shell. Reports what constrains a change, not what a subsystem contains -- patterns to follow, utilities that already exist so they are not rewritten, non-obvious breakage. Told to state what its slice did NOT cover, so gaps are visible to the synthesiser instead of hidden by implied completeness. - agents/uw-critic.md: Read, Grep, Glob. Attacks a draft test plan from one angle. Its question is not "are these good tests" but "could an implementation pass every one of these and still be wrong". - workflows/plan.js: three fixed readers -- code, project docs, memory topicals -- plus one per named area, then a barrier and a synthesiser. The barrier is required: a design review is a claim about how pieces fit, which nobody holding one piece can make. - workflows/tests.js: draft, attack from four angles, merge. The author of a test plan is its worst judge, because a test that checks what they meant looks complete to them. Two prompts carry most of the value. The merge stage may decline a gap and must say why, because appending every suggested test produces a plan nobody runs; it is told to prefer strengthening a weak assertion over adding a case. And oracle-quality is called out as the sharpest angle: a test asserting only that something returns, or does not throw, passes against a broken implementation and stops anyone looking further. Untested against a real run -- uw-explorer and uw-critic were added this commit and agent definitions load with a delay. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shell-write heuristic added an hour ago matched bare heredocs, and every `git commit -F -` uses one. From the moment it shipped, the stop gate demanded a test run after every single commit -- which the gate then did, blocking a turn one commit later. Its own comment said "over-marking costs one test run". Once per commit is not one test run, it is a permanent tax, and that is how a gate gets switched off. The rule was also wrong on its merits: `cmd <<EOF` feeds stdin, not a file. Writing a file needs `cat > file <<EOF`, which the redirection rule already catches. So it contributed false positives and no coverage. Two regression tests pin both directions: a heredoc commit message is not a write, a heredoc redirected into a file still is. Found by living under the gate, not by the tests -- both fixes passed 148 green tests before this surfaced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The only phase that writes, and the only one with a loop. Both properties are why it was built last. - agents/uw-implementer.md: the sole agent with Write and Edit. By the time work reaches it, a plan and a test plan are both confirmed, so its job is to execute them, not redesign them. - workflows/build.js: write tests, confirm they FAIL, implement, verify, loop. Three decisions that make the loop safe rather than a way to burn tokens: Tests must fail first, and the run stops if they do not. A test that passes before the implementation exists is testing nothing, and it is invisible afterwards -- a green suite looks identical whether its tests constrain the behaviour or merely coexist with it. Implementing against already-passing tests would go green on round one and prove nothing. Diagnosis happens in a fresh context, by uw-reviewer rather than the implementer. The agent that just failed is holding its own wrong theories; a reader with only the failure and the diff sees what the evidence actually says. This is also what keeps raw stack traces out of the conversation. Two stop conditions, not one. The hard cap at three rounds, and an earlier exit when the same failure signature repeats -- no progress is a reason to stop before the cap, not after it. Escalation follows the blocker protocol the workflow already specifies: the problem, what changed between attempts, the signature, and options rather than a pick. The implementer is told never to make a test pass by weakening it, because that converts a failing test into a lie that outlives the session. Untested against a real run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three false positives, zero true positives, so the heuristic is gone rather than tightened again: a heredoc (every `git commit -F -`), an arrow function (every `node -e` one-liner), and `2>/dev/null` (ordinary stderr suppression, which writes nothing). Its own comment claimed "over-marking costs one test run"; it cost one after nearly every command, which is how a gate gets switched off rather than fixed. The gap it aimed at is real -- a shell can write source. But inferring intent from a command string is the wrong instrument, and tool scoping answers it structurally: an agent without Bash cannot route around anything. A test now pins the removal, asserting that none of the three commands that used to trip it mark the tree dirty. Also adds research/, with Google's May 2026 paper "The New SDLC With Vibe Coding" and a summary, indexed from memory.md so it loads on demand rather than sitting in always-on context. The paper independently describes most of this plugin from the outside -- "Agent = Model + Harness", and "most agent failures are configuration failures" -- and cites the Terminal Bench result that motivated the graph-native rebuild. It also settles an argument we had left open: the static/dynamic context boundary should be "reviewed and versioned like any other configuration", which we have never done. Our static half only grows and our dynamic half was empty until this commit. Where we know more than the paper: it treats hooks as reliably deterministic. We measured three routes to a hook being installed and silently enforcing nothing. Its harness component list has no entry for verifying the harness is running, and on this evidence it should. 147 tests, 144 pass, 3 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The five-step order held. The definition of done did not. The paper draws a line this plan had no word for: tests verify the deterministic parts, evals verify the non-deterministic ones -- trajectory, tool choice, output quality -- and "without both, the practice is always vibe coding, regardless of how sophisticated the prompts are." We have 147 tests, every one covering a pure predicate, and zero evals. Nothing checks whether /review picks the right lenses, whether /harvest classifies Gotcha-versus-Memory correctly, or whether the verifier's refusals are sound. The two runs that convinced us those phases work were demos, not evals -- exactly the distinction the paper draws for engineering leaders. By this plugin's own standard, the phases are vibe-coded. Shipping them as agentic engineering would be selling a discipline we are not practising. So an eval harness moves from follow-up to precondition. Three smaller gaps from the same source: no model routing despite agent() taking model and effort (/review cost 460k tokens, /plan 481k); a static/dynamic context boundary that has never been reviewed, where the always-on half only grows and memory/ was empty until today; and observability that answers "did a gate run" but not "is this drifting". Revised order: eval harness, then model routing with cost recording, then run /tests and /build against the evals rather than a demo, then the plan-file ambiguity, then the context boundary, then ship. Also records what living under the gates taught: the deleted heuristic, the escaping failure that corrupted content four times, the plan-file ambiguity that locked out a bug fix, and the corrected claim about agent definitions loading with latency rather than needing a restart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six defects surfaced while planning, none previously known.
D1 The manifests never registered what the plugin actually ships. All three
still said version 2.4.0 and described it as "Two skills", with no mention of
the five workflows or five agents on disk. Bumped to 3.0.0 with a description
that matches reality.
D2 The bootstrap skill's own front matter said it produces "three bootstrap
docs" while its body produces four; memory.md was added and the description
never followed. Fixed in both mirrors.
D3 progress-template.md shipped a literal `## YYYY-MM-DD` heading, and the
bootstrap handoff predicate at state.js:133 requires a real
`^## \d{4}-\d{2}-\d{2}`. A project using the template as-is would never hand
off. Replaced with a real example date and a note explaining why it must stay
one -- a template that cannot pass the gate it feeds is a trap for whoever
uses it first.
D7 memory.md's only entry pointed at research/, contradicting the memory/
convention the same file documents two lines above. Marked as an external
reference rather than silently breaking its own rule.
D8 Three PLAN-*.md at the root made readState's mtime selection ambiguous,
which locked out a bug fix earlier in this session. The two completed plans
move to docs/plans/, leaving exactly one active. Their content is preserved
for the memory/ topicals still to be written.
D10 .claude/agents/uw-harvester.md had drifted from agents/uw-harvester.md --
the local copy lacked the stale-audit paragraph. That paragraph was added
after the runs recorded as validating the harvester, so the local dogfood
copy was behind the shipped one and that behaviour was never exercised.
Reconciled.
147 tests, 144 pass, 3 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cursor has no `tools` key -- tool access is inherited from the parent -- but it does have `readonly: true`, which restricts write permissions. That is the same write-isolation property under a different name. So each read-only agent now carries both keys. Claude Code reads `tools:` and refuses anything outside it; Cursor reads `readonly:` and blocks writes. Each host ignores the other's key, so one file serves both and there is no second copy to hand-maintain. uw-implementer deliberately carries neither -- it is the only agent that writes -- and a check asserts it never gains `readonly`, because a stray copy-paste there would silently disable the only phase that can change the repository. This replaces the approved plan's `phases/<name>.md` idea, which cannot work as specified: workflow scripts have no filesystem access, so a script can never load a phase file. The agent definition already is the single source -- it loads automatically when the agent spawns, on both hosts -- so the duplication is removed by trusting it rather than by adding a directory. Cursor also reads `.claude/agents/`, so the plan's separate `.cursor/agents/` copies are unnecessary. Frontmatter comments were tried and removed: they are valid YAML but parsers vary, and a definition that fails to parse fails as a missing agent type, which is silent until a workflow errors. The explanation moved into the body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gap the Day 1 paper named: tests cover the deterministic parts, evals cover the parts that are not -- trajectory, tool choice, retrieval. Without both, the practice is vibe coding regardless of prompt quality. This plugin sells that standard, so it has to meet it. evals/lib/score.js scores facts, not prose: did it open this file, did it touch those lines, did it surface that constraint. That choice is what keeps the signal above the noise of non-deterministic agents -- two identical /harvest runs in this repo produced different proposals, so anything scored on output quality would be measuring variance. Two of the eight fixtures pass by reading NOTHING extra. Without those negative controls the eval rewards over-retrieval, which scores well while undoing the entire point of moving content out of always-on context. compare() names section-level rollback in its own verdict string: if thin loses a fixture, restore the section that fixture covers rather than reverting the change. Deciding that in code now rather than under pressure later. Known weakness, stated in the runner rather than hidden: trajectories are self-reported by the agent, not observed by the harness. An agent that misreports which files it read defeats this. That is why the verdict claims only "no detected regression" and nothing stronger. 15 scorer tests. 162 total across the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured before touching anything: 58% of CLAUDE.md was rarely relevant. 1220 of its 2769 words were byte-identical to SKILL.md; 715 more were memory-system guidance consumed at most once per feature; 427 were Gotchas that only matter when touching one subsystem. Guidance about where to write things down, plus the things written down, came to 49% of the file -- larger than the workflow steps it served. The rule that decided every move: keep what has no cue, move what has a cue. Static context is enforced by presence, dynamic by retrieval, and retrieval fails when the trigger is an absence. "Do not add a dependency" has no file glob; it stays. "Cursor prefixes payloads with a BOM" has an obvious one; it moves. Four memory/ topicals now carry the subsystem knowledge, each indexed with a Read-when cue. references/memory-protocol.md takes the once-per-feature guidance -- its real consumer was already workflows/harvest.js, which inlines the decision test into the router prompt, so keeping 715 always-on words was paying twice for the second-best copy. memory/context-boundary.md is mandatory rather than nice-to-have: it opens by telling the next session that a thin CLAUDE.md looking incomplete is the failure it exists to prevent. The restructure has to defend itself. Cursor's structure fell out simpler than planned. Its alwaysApply rule is its CLAUDE.md, so it becomes a thin index too -- and since Cursor also reads skills/, the full body stops being duplicated at all. Three mirrors become one pair of small files. The rule states plainly what Cursor loses: ordering is advisory there, the round cap is a number you honour rather than one that is counted. The curl install path now fetches SKILL.md. A thin index has dead pointers for someone holding a single file. mirrors.test.js replaces the README's honour-system note. That note existed and an undocumented divergence crept in anyway -- a documented divergence list without a check is a comment, not a control. It also asserts the caps (the file will not stay thin on its own), that every pointer resolves, that every index entry carries a Read-when cue, and that exactly one agent can write. 174 tests, 171 pass, 3 skipped. Fat baseline captured in evals/results/ before the change; thin run pending. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gates were the enforcement story in 2.4.0. They are not any more: the phase commands make ordering structural, so a phase that has not been started cannot be skipped and there is nothing to intercept. What is left for the gates is ad-hoc work outside the phases -- a backstop. The demotion is measured, not stylistic. Full enforcement on one host, partial on one, unverified on two, and three separate routes to "installed and silently enforcing nothing". A default-on mechanism that fails open invisibly is worse than an opt-in one somebody chose. ULTIMATE_WORKFLOW_GATES now turns gates ON (on/1/true/yes); absent means off. gatesDisabled stays as the exact inverse so either caller agrees. The end-to-end tests now opt in explicitly, which asserts the opt-in path rather than relying on a default that just changed. Deliberately separate from the context restructure: the two confirmation points survived on always-loaded prose PLUS gates, and landing both at once would remove both reinforcements in a single commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
score.js had the API but nothing to score against -- the Part 5 pass conditions existed only in the plan. They are now in evals/expectations.js, separate from context-boundary.js because that is a workflow script with no require, no filesystem, and a top-level return, so nothing can import it. expectations.test.js checks the two lists agree, since a forced split with no check is a comment rather than a control. It also rejects a fixture with zero assertions and an expectation key score() would silently ignore. thin.json records all 8 trajectories. Scoring both arms gives fat 4/8 and thin 8/8 with no regressions, and results/README.md explains at length why that number should not be quoted: three of four fat failures are files that did not exist under fat (the topicals landed in the same commit that thinned CLAUDE.md), and the mustMention conditions were authored with the thin run already visible. Only C5's mustRead was specified in advance. What this supports is the plan's own claim and nothing stronger: no detected regression on C2, C3, C4, C7, one run per cell. thin.json is useful from here as a fixed baseline, which is the smaller real value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README.md, SKILL.md and both always-on indexes all advertise /ultimate-workflow:plan and its four siblings. Nothing implemented them. workflows/*.js are scripts for the Workflow tool -- memory/workflows-authoring.md says so plainly, "invoke the Workflow tool with scriptPath" -- and a plugin's command surface is commands/*.md, which this repo did not have. Each command is a thin shim that calls its script, plus the part that cannot live in a script: what to do with the result. plan and tests both end in an explicit stop, because the confirmation is the product and neither plan_confirmed nor tests_confirmed is the model's to set. review is told to list refuted findings rather than drop them. harvest applies nothing. mirrors.test.js gains four checks so the surface cannot drift: every phase has both halves, no command names a missing script, no script is unreachable, and every command declares a description. An advertised feature that is not there is the exact defect class this release keeps turning up. Also fixes the handoff regex in progress-template.md, where the backslashes had been eaten -- the same escaping trap already recorded in Gotchas, this time in prose rather than code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leads with why the enforcement story changed rather than with the feature list: seven of nine defects in 2.4.0 were in the vendor wire surface and every one failed open and silently. Records the inverted sense of ULTIMATE_WORKFLOW_GATES prominently, since anyone relying on the old default gets nothing without noticing. Known limitations carries the measured fan-out cost, the fact that the eval baseline is one run per cell, and that two of four hosts have never been exercised against a real payload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The topicals were unreachable on Cursor and nothing said so. On Claude Code a topical is reached through its Read-when cue in memory.md; Cursor does not act on that pointer. Its dynamic-context mechanism is an alwaysApply:false rule selected by description and globs, so a pointer list in an always-on rule is not a cue Cursor ever follows. Four topicals, four rules, none of them previously existing -- the memory/ half of this release was Claude Code only in practice while reading as cross-host in the docs. Each rule is a selector, not a copy: it fires on the topic, states why the read is worth it, gives the two or three moves you make before touching anything, and sends you to the topical. A fifth hand-maintained copy of the same prose is the drift the rest of mirrors.test.js exists to prevent, so a test asserts each rule stays shorter than what it points at. Five more checks keep the sets in step: every topical has a rule, no rule names a missing topical, every rule is actually on-demand, every description is long enough to select on (a vague one silently never fires), and the always-on rule stays under its line cap as the on-demand set grows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v3 dogfood. workflows/tests.js and workflows/build.js had been written and never once executed, so this release was shipping two phases nobody had run. Both now have. /tests: draft, four critics in parallel, merge. 6 agents, 215k tokens. It turned a 19-test draft into 18 stronger ones and declined three suggestions with reasons. The coverage and edge-case critics independently found the same hole -- no fixture set dirty: true, so an implementation deriving `blocking` from preEditGate alone and never calling stopGate passed the entire draft while contradicting the human report on the most common real state. /build: wrote the tests, confirmed 14 of 18 FAILED for the right reason before implementing, then green on round 1. 2 agents, 115k tokens. The feature itself follows the compute/render split the rest of hooks/lib already uses: one `report()` pass, two renderers over it, so the human and machine outputs cannot disagree. The escape hatch delegates to stateLib.gatesDisabled rather than testing the variable inline. Also fixes a live defect in committed code, found by the runnability critic while it was only asked whether a test plan could run: status.test.js used execFileSync, which returns stdout alone and forwards the child's stderr to the parent, so r.stderr was undefined and every stderr assertion in that file was vacuous -- including the one claiming to prove no stack trace escapes. Now spawnSync. Test 15 skips on unprivileged Windows (symlink), by name, and runs on CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ran /review over the whole branch: four lenses, then an adversarial pass that refuted two of fourteen findings. Two of the twelve confirmed are severe, and both were reproduced end to end by the verifier before being accepted. G4 scored a FAILING verification run as green. Pass/fail came from the event name alone, on the belief that a failure arrives as postToolFailure. Claude Code does not appear to send it -- fixtures/claude-code/PostToolUse.18.json is a non-zero-exit command recorded as a plain PostToolUse, and no failure fixture exists among the recorded payloads. So a red run wrote last_verify: green, cleared dirty and reset the round counter: failing the tests was the way to satisfy the gate that exists to make you pass them. normalize() already derived `ok` from tool_response.success and nothing read it; now any signal meaning failure means failure. G3 never surfaced a recorded gate_violation -- the entire compensation for a host that cannot block a write -- because it was checked only after stopGate's verdict, and stopGate steps aside when no verification command is set. The shipped plan template ships test_command: "". So on Cursor, the one host that compensation exists for, G1 and G2 were bypassed and the violation was written to the plan and shown to nobody. status.js now lists it too. Both regression tests were confirmed to FAIL against the pre-fix code. The rest: pathMatches matched any same-named file in any directory, which corrupts the eval's negative controls in both directions; C7's oracle scored the ideal answer as a failure and the vague one as a pass; the only test touching the eval baselines asserted typeof pass === 'boolean', true for every input, so the documented numbers were computed by nothing; the hookEventName regression test rebuilt the buggy line by hand instead of running gate.js, and reverting that line left the suite green; hooks/hooks.json, which decides whether any gate runs at all, was read by no test and not even parsed by CI; a verbatim duplicate test and an exception table holding zero exceptions. hooks/registration.test.js is new and mutation-checked: dropping the Stop block turns it red. The 9.97 MB paper shipped inside every plugin ZIP -- the file Desktop and offline Cursor users download -- for a binary nothing reads. Excluded, with a 2 MB ceiling in CI so the next large addition fails loudly. --json is now documented in the README, since it shipped as a public contract whose only definition was an object literal and a gitignored plan file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ran /harvest over the branch: three readers, then a router. 10 candidates, 7 proposals, 4 discarded -- including declining to invent a progress.md this repo does not have, on the grounds that CHANGELOG.md and the commit messages already carry that story and a third copy would need keeping in sync. The most valuable output was the staleness audit, not the new entries. memory/hooks-and-gates.md asserted as measured fact -- "all except Gemini fire a distinct tool-failure event" -- the exact belief that inverted G4. The topical every host and adapter task is cued to read was standing the bug up as a warning. Corrected, keeping the true half. memory/mirrors.md promised a control that no longer exists: "edit both in the same commit, mirrors.test.js fails otherwise" is false for both skill pairs, since the .mdc stopped being a body mirror when it became a thin index. A reader following it would edit one side and trust a green suite -- precisely the failure that file was written to prevent, now caused by the file. Both always-on indexes said to verify with a command that omits mirrors.test.js, which sits at the repo root where neither glob reaches it. The documented local command skipped the mirror check and went green while CI went red. New: don't assert on stderr from execFileSync (Gotcha, both mirrors); a compensating control must not sit behind the control it compensates for; and the multi-plan mtime ambiguity, which is unfixed and live in this root right now. The cost table finally has all five phases, because all five have run. The backslash Gotcha goes to hit 5x -- a heredoc ate one again this session, in the very test written to prove path matching was strict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It told the reader to unset ULTIMATE_WORKFLOW_GATES to re-enable, which was true when gates were on by default and is now exactly backwards -- the one line someone reads at the moment they are trying to turn gates on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old README was written for someone who already knew the project. It led with "lifecycle gates" and "context compaction", spent thirty lines on a JSON contract and hook payload internals, and never plainly said what the idea is. The concept now gets its own section in ordinary words: stop asking one assistant to do everything in one long conversation, split it into five short jobs that each start fresh, do one thing, and stop. The three consequences follow from that -- it cannot skip a step because the next step does not exist until you start it, it cannot wander because a reading job only gets reading tools, and it cannot lose the thread because each job hands back a file rather than a memory. The cost is stated in the same breath: five commands instead of one, and you move between them. Also says plainly that you do not need all five every time. The gate internals, the --json contract, the per-host measurements and the two gate-inverting defects move to docs/gates.md rather than being dropped. That material is the honest part and it is worth keeping; it just is not the front door. The README keeps a four-row table of what each host actually gives you and a one-line pointer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deleted hooks/no-emoji-write.js and hooks/no-emoji-prompt.js and removed their two registrations from hooks.json, which now registers gate.js and nothing else. The rule survives once per host, in CLAUDE.md and the Cursor always-on rule, alongside the other hard constraints. Also narrowed in scope, from "no emojis anywhere" to "no emojis in code". Code is where the breakage actually is -- a terminal rendering a source file, a pipeline parsing output. Prose was never the problem, and the blanket version put the project in the position of enforcing against its own public page, which carries four principle icons by design. The registrations were removed structurally rather than by string surgery. hooks.json decides whether any hook runs at all and CI only started parsing it this release; a stray comma there is silent. README no longer claims the plugin refuses to write emoji into your files, because it no longer does. mirrors.test.js asserts the new wording, so both always-on files still have to carry the constraint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/index.html is what the README calls the "Visual walkthrough" and it had drifted an entire major version. Content and structure only -- the CSS, the card system, the layout and the principle icons are untouched. One item was a live bug rather than staleness: the page told readers to run `/plugin install the-ultimate-workflow-guidelines`, a name changed two releases ago in 2.4.0. That command fails. Two were actively harmful. The page described CLAUDE.md as mirroring the skill body and rules/*.mdc as "same body content" -- precisely what v3 stopped doing, and precisely what memory/context-boundary.md exists to prevent. A reader following the public page would have re-inlined the body and undone the restructure. The rest is the v3 story it never told: the five-jobs concept and the three things that follow from it, the five commands and which one can write, an honest per-host table, and a link to the gates document. Internal planning notes move from docs/plans/ to internal/plans/. GitHub Pages serves docs/, so merging this branch would have published two candid working documents that were never written for strangers. Source comments citing them now resolve, and the release ZIP excludes internal/ instead. Five checks added to mirrors.test.js, each confirmed to fail against the defect it describes: every phase is named and the count matches the scripts, no install command names anything but the manifest's plugin name, the site and README claim the same hosts, the body-copy claims stay gone, and nothing under docs/ is a PLAN file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A user-visible removal, so it gets its own Removed section rather than a line in Changed: nothing blocks an emoji write any more and ALLOW_EMOJIS=1 is inert. Says plainly that anyone relying on the hook no longer has it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
main shipped AGENTS.md as the cross-tool root doc, /handoff, and a version renumber while this branch was building v3. Eleven files conflicted. The one real decision: both lines independently solved "the always-on file is too big", in incompatible ways. main moved the content to AGENTS.md, which Cursor reads natively and Claude Code imports through a one-line CLAUDE.md stub. This branch made the always-on file a thin index pointing at memory/ topicals. They combine: AGENTS.md keeps main's cross-tool convention and takes the thin-index treatment, 241 lines to 60. Measured first -- 90% of its lines were byte-identical to SKILL.md, the same finding that thinned CLAUDE.md. A functional bug the merge would otherwise have shipped: handoffComplete read CLAUDE.md for `## Key files` and `## Gotchas`, and the new stub is one line with no headings. Every newly bootstrapped project would have been permanently un-handed-off, with the gates blocking feature work and pointing at a file that can never satisfy them. Now either file may carry the sections, so projects bootstrapped before the rename keep working. Three tests cover it. The CHANGELOG's [2.4.0] never shipped -- main used that number for other work and renumbered it to 2.6.0 -- so it is folded into 3.0.0 rather than stacked, and the 3.0.0 narrative no longer credits a release that does not exist. The plugin rename becomes a 3.0.0 breaking change. Carried forward from main: truth-over-agreement, never-fake-success, /handoff as workflow step 7, the templates pointer, the ref_name fix and the AGENTS.md zip exclusion. Kept from this branch: the phases, gates, evals, agents, the Cursor retrieval rules and the rewritten README and site. Also fixed: main's bootstrap description still undercounts its own output, omitting memory.md -- the same defect this branch fixed, in new wording. The README's CURSOR.md link is dead, since main folded that file into the README. mirrors.test.js retargets to AGENTS.md, asserts CLAUDE.md stays a bare import, and allows a prompt-only command such as /handoff to name no workflow script. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
v2.4.0 answered "the model ignores the workflow" with enforcement: five lifecycle hooks that block a bad edit. This release keeps them but stops leading with them, because measuring them changed the answer.
Seven of the nine defects found in that work were in the vendor wire surface, not the logic, and each failed open and silently. Three separate routes led to "installed, enforcing nothing, looking identical to working".
The alternative turned out to be structural rather than defensive. If each phase is its own invocation with its own fresh context and its own tools, ordering is not policed - a phase you have not started cannot be skipped. That is what this release is.
Highlights
plan,tests,build,review,harvest. Each runs in a fresh context, does one job, and stops. Onlybuildcan write; the other four return proposals you apply.ULTIMATE_WORKFLOW_GATES=on), landed in its own commit because the two confirmation points previously survived on always-loaded prose plus gates.CLAUDE.mdis a thin index, 240 lines to 49. 1220 of its 2769 words were byte-identical toSKILL.md.memory/topicals were unreachable there - Cursor does not follow a pointer list; it selectsalwaysApply: falserules by description. Four rules added, each a selector rather than a copy.evals/- a trajectory scorer plus eight context-boundary fixtures, the gap named by Google's Day 1 paper: tests cover deterministic behaviour, nothing covered trajectory or tool choice.ALLOW_EMOJIS=1is inert. See the CHANGELOGRemovedsection.It was dogfooded on itself
All five phases ran against this repo.
/testsand/buildhad never been executed before; they built thestatus.js --jsonflag end to end, including proving the tests failed first./reviewreturned 12 confirmed findings after an adversarial pass refuted 2. Two inverted the gates, and both were reproduced end to end before being accepted:postToolFailure. Claude Code does not send it - a recorded fixture shows a non-zero-exit command arriving as plainPostToolUse. A red run wrotelast_verify: greenand reset the round counter, so failing the tests was how you satisfied the gate that exists to make you pass them.gate_violation- the entire compensation for a host that cannot block a write - because it sat behindstopGate, which steps aside when no verify command is set, and the shipped template shipstest_command: "".Both regression tests were confirmed to fail against the pre-fix code.
/harvest's staleness audit then found thatmemory/hooks-and-gates.mdasserted as measured fact the exact belief that caused the G4 inversion.Documentation
The public page had drifted a full major version. One item was a live bug: it documented
/plugin install the-ultimate-workflow-guidelines, a name changed in 2.4.0, so the command fails. Two were actively harmful - it describedCLAUDE.mdandrules/*.mdcas body copies, which is what v3 stopped doing.Internal planning notes moved out of
docs/, which GitHub Pages serves; merging as-is would have published them.Five checks were added to
mirrors.test.jsso the site cannot rot silently again. Each was mutation-tested against the defect it describes.Verification
ubuntu-latest- this PR is the first time it executesKnown and stated, not hidden
/harvest158k tokens,/review460k,/plan481k.evals/results/README.mdargues against its own headline number at length.🤖 Generated with Claude Code