fix(development-pr-workflow): backtest-change also triggers for NEW monitors/gates - #555
Conversation
…onitors/gates The command only advertised itself for CHANGES to existing data-driven config, so authoring a brand-new gate monitor with a backtest-derived threshold (the INF-335 EGW-Go blue/green gate) didn't trigger it even though the methodology is identical. Add new-monitor/gate trigger language and mention joint rate-AND-floor replay for gate composites.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📚 Documentation Check ✅Verdict: Passed The only plugin touched (development-pr-workflow) had its version bumped in both packages/plugins/development-pr-workflow/.claude-plugin/plugin.json and the root CLAUDE.md version table (2.4.0 → 2.4.1), satisfying the blocking requirement. No other docs are stale as a result of this change. Findings\n\nVersion bump: done.
|
|
● Reviewed · against Caution Changes requested — the skill's YAML frontmatter no longer parses, so Broadens the AssessmentThe scope expansion is consistent with the skill body — step 4 already handles "old is 0" for new monitors, and the composite joint-replay paragraph matches it. But collapsing Must address
Suggestions
Tip Teach the reviewer. React 👍 on findings that helped, 👎 on false positives. Reply to push back or add context — we aggregate this weekly to tune the bot. Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
📋 Review verdict: COMMENT
👆 The main review comment above is the source of truth for this PR review. It is automatically updated on each review cycle, so always refer to it for the most current feedback.
This formal review submission is for the verdict only.
…kill The base branch converted the backtest-change command into a thin wrapper over a skill, so the trigger wording this branch broadened now belongs on the skill's description — that is what auto-triggers. - Take next's command description (no longer drives triggering) - Skill description: fires for brand-new monitors/gates, not just edits - Replay step: composite gates must be replayed jointly, since per-condition counts overstate what the gate does - Bump development-pr-workflow 2.4.0 -> 2.4.1 (+ root CLAUDE.md table)
Matches the dominant repo convention (40 of 43 SKILL.md files use a single-line description; only 3 used the folded scalar). Addresses review feedback on the mid-sentence line breaks. Text unchanged.
| throughput target. A proposed number is a hypothesis, not a decision: backtest | ||
| it and let the data override it. Always report old N vs new M with the window | ||
| and data source. The /backtest-change command loads this same skill. | ||
| description: Validate a data-driven change against LIVE historical data before it ships — replay old-vs-new over a real window, report whether it achieves its goal, and refuse to ship when the data disproves the premise. Fires whenever someone proposes a measurable change and names a number: "add a monitor at 700MB", "set the threshold to N", "warn at X / critical at Y", "alert when it exceeds N", "raise the timeout to 5s", "change the sampling rate", "bump the cache TTL", "tighten this alert", "loosen the threshold", "this should reduce the noise", "that will fix the p95" — and before opening any PR for a monitor threshold, alert routing or renotify cadence, metric/log/trace query, sampling rate, rate limit, autoscaling parameter, or a perf change with a latency or throughput target. Fires for a brand-new monitor, deploy gate, or alert just as it does for an edit to an existing one — a threshold invented for a gate that has never run needs the same historical replay, including the joint replay of every condition in a composite (e.g. rate AND floor together, not each in isolation). A proposed number is a hypothesis, not a decision: backtest it and let the data override it. Always report old N vs new M with the window and data source. The /backtest-change command loads this same skill. |
There was a problem hiding this comment.
Caution
correctness · frontmatter no longer parses
Collapsing the description: > folded block into an unquoted plain scalar puts : inside the value (names a number: "add a monitor…"), which YAML rejects — yaml.safe_load fails with mapping values are not allowed here at line 3, col 292. The skill's name/description/allowed-tools/model are all unreadable, so it won't register or auto-trigger for anything — the inverse of this PR's goal. Single-quoting fixes it (the text contains no apostrophes); restoring description: > also works.
| description: Validate a data-driven change against LIVE historical data before it ships — replay old-vs-new over a real window, report whether it achieves its goal, and refuse to ship when the data disproves the premise. Fires whenever someone proposes a measurable change and names a number: "add a monitor at 700MB", "set the threshold to N", "warn at X / critical at Y", "alert when it exceeds N", "raise the timeout to 5s", "change the sampling rate", "bump the cache TTL", "tighten this alert", "loosen the threshold", "this should reduce the noise", "that will fix the p95" — and before opening any PR for a monitor threshold, alert routing or renotify cadence, metric/log/trace query, sampling rate, rate limit, autoscaling parameter, or a perf change with a latency or throughput target. Fires for a brand-new monitor, deploy gate, or alert just as it does for an edit to an existing one — a threshold invented for a gate that has never run needs the same historical replay, including the joint replay of every condition in a composite (e.g. rate AND floor together, not each in isolation). A proposed number is a hypothesis, not a decision: backtest it and let the data override it. Always report old N vs new M with the window and data source. The /backtest-change command loads this same skill. | |
| description: 'Validate a data-driven change against LIVE historical data before it ships — replay old-vs-new over a real window, report whether it achieves its goal, and refuse to ship when the data disproves the premise. Fires whenever someone proposes a measurable change and names a number: "add a monitor at 700MB", "set the threshold to N", "warn at X / critical at Y", "alert when it exceeds N", "raise the timeout to 5s", "change the sampling rate", "bump the cache TTL", "tighten this alert", "loosen the threshold", "this should reduce the noise", "that will fix the p95" — and before opening any PR for a monitor threshold, alert routing or renotify cadence, metric/log/trace query, sampling rate, rate limit, autoscaling parameter, or a perf change with a latency or throughput target. Fires for a brand-new monitor, deploy gate, or alert just as it does for an edit to an existing one — a threshold invented for a gate that has never run needs the same historical replay, including the joint replay of every condition in a composite (e.g. rate AND floor together, not each in isolation). A proposed number is a hypothesis, not a decision: backtest it and let the data override it. Always report old N vs new M with the window and data source. The /backtest-change command loads this same skill.' |
| { | ||
| "name": "development-pr-workflow", | ||
| "version": "2.4.0", | ||
| "version": "2.4.1", |
There was a problem hiding this comment.
🔵 info · plugin-conventions — The diff also adds a workflow requirement to step 4 (joint replay of composite conditions), which reads as a backward-compatible capability under this repo's minor-bump criteria; consider 2.5.0 instead of 2.4.1. The PR body's "command body untouched" is also inaccurate — SKILL.md step 4 gained four lines.
There was a problem hiding this comment.
Caution
2 participants
Description
/backtest-changecommand's description only advertised itself for changes to existing data-driven config ("Before opening a PR for a data-driven change…"), so authoring a brand-new monitor with a backtest-derived threshold never triggered it — even though the methodology is identical. Concrete miss: the INF-335 EGW-Go blue/green gate PR (Uniswap/backend#11307, 2026-07-30) ran the exact backtest-change workflow (21d live-history replay, threshold headroom statement) manually because the command's trigger language didn't cover new-monitor authoring.Testing