PPLT-5844: add IntelliStory storybook affected-story filtering - #2339
Conversation
…iltering Introduces IntelliStory: given a Storybook build, it diffs against a baseline (explicit or API-predicted) and filters the snapshot set down to the stories a change actually affects, bailing to a full run whenever it can't reason about the change. - cli-command: intelliStory.js (applyIntelliStory, IntelliStoryBailError), lockfileDiff.js and graphTrace.js, exported via the ./intelliStory subpath; adds glob-to-regexp, stream-json and optional snyk-nodejs-lockfile-parser. - client: getStatus() accepts the intelli_story_graph job type, plus getIntelliStorySnapshotNameToCommit() and generateIntelliStoryGraph() hitting the /intelli_story endpoints. Binds createRequire to cjsRequire (not `require`) in intelliStory.js and lockfileDiff.js so the CommonJS-transpiled packaged binary doesn't crash with "_require is not a function", and adds noRequireBinding.test.js as a static regression guard (with the matching .semgrepignore rationale).
…trim comments - intelliStory.js: read the enriched stats file with a single JSON.parse(fs.readFileSync(...)) instead of streaming it with stream-json. The stats payload is only a few MB, so in-memory parsing is simpler and removes the stream-json dependency (and, with it, the last createRequire binding in this file — the packaged-binary footgun now only exists in lockfileDiff.js for the optional snyk parser). - Remove stream-json from cli-command dependencies and the lockfile. - Strip explanatory comments introduced with the feature across the IntelliStory source, tests and trace template. Coverage directives (istanbul ignore), semgrep suppressions (nosemgrep) and the cjsRequire / loadSnyk rationale in lockfileDiff.js are kept — removing them would break the 100% coverage gate and the semgrep scan.
Add tests for the graceful-bail paths added in "apply suggested fixed": - malformed stats-file JSON (validateAndReadStats) - unreadable current lockfile (getAffectedPackages) - unreadable package.json (getAffectedPackages) These run on Node 14 (they bail before the Node>=18 snyk path), fixing the lines/statements coverage-threshold failure on the Node 14 test job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GLOB_CHARS only detected * and ? so brace ({a,b}) and bracket ([abc])
patterns fell through to exact-match, silently disabling the intended
glob semantics for these documented config options.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude Code PR ReviewPR: #2339 • Head: 001c036 • Reviewers: stack:code-reviewer SummaryAdds IntelliStory — an affected-story graph feature for Review Table
Findings
Verdict: PASS |
Claude Code PR ReviewPR: #2339 • Head: 966c05d • Reviewers: SummaryAdds IntelliStory to The bail-first posture is the right architecture for a feature that can silently drop visual coverage, and it is applied thoroughly — 15+ explicit bail sites covering missing build id, unreadable/invalid stats, git failures, ambiguous or absent lockfiles, manifest changes spanning directories, browser-upgrade builds, and graph-job timeouts. All four Medium findings below are gaps in that otherwise-sound safety net, not architectural problems. Review Table
Findings1. API/network failures escape as raw errors instead of
|
Claude Code PR ReviewPR: #2339 • Head: 01aa1f6 • Reviewers: SummaryAdds IntelliStory to
Review Table
Findings1.
|
Claude Code PR ReviewPR: #2339 • Head: 91bd770 • Reviewers: SummaryAdds IntelliStory to This head resolves the High finding from the Fix verification (vs.
|
| # | Finding | Sev | Status at 91bd770 |
|---|---|---|---|
| 1 | bailOnChanges/untraced globs not rebased to the invocation dir |
High | Fixed |
| 2 | Client calls throw raw Error, not IntelliStoryBailError |
Medium | Open |
| 3 | .storybook config dir hardcoded |
Medium | Fixed |
| 4 | Malformed glob silently swallowed to "no match" | Medium | Fixed |
| 5 | Dropped stats modules logged at debug, no threshold |
Medium | Fixed |
| 6 | Bare-extension globs don't match nested paths | Medium | Open (by design; still undocumented) |
| 7 | noRequireBinding regex narrower than the crash class |
Low | Open |
| 8 | mapEntry normalizes source only for type === 'src' |
Low | Open |
| 9 | gitDiffNames omits --no-renames |
Low | Open |
| 10 | path.relative does not case-fold |
Low | Open (was unconfirmed) |
| 11 | trace.html loads Drawflow from a CDN |
Low | Open |
| 12 | No log.warn at bail throw sites |
Low | Partially addressed |
Finding 1 is fixed properly, not papered over. resolvePattern (intelliStory.js:88-100) rebases each user pattern from the invocation directory onto the repo root — the same basis normalizeImportPath already used for story paths, so the asymmetry that caused the bug is gone. Beyond that, the fix handles the two readings asymmetrically by direction of harm, which is the right call:
assertNoBailOnChanges(:394-425) bails when a pattern matches only under the repo-root reading, logging a warning with a concrete rewrite suggestion — a stale root-relative config still protects the user.enforceUntraced(:433-462) declines to apply such a pattern and warns, since applying it would drop snapshots.
A <rootDir>/ prefix is an explicit opt-out, patterns resolving outside the repo bail rather than matching nothing, and resolveConfigDirs (:361-390) additionally checks that a configured directory actually exists — catching a wrong-basis configDir at config time instead of silently never matching. Findings 3, 4 and 5 are each fixed with matching care: isInsideDirs distinguishes single-segment (any depth) from multi-segment (prefix) config dirs so config/storybook can't match config/storybook-old; patternError validates globs up front and routes a compile failure to a bail for bailOnChanges and a warn-skip for untraced; and readStats now separates deliberately-excluded modules from unresolved ones, warns on the latter and bails past a 10% ratio of resolvable candidates — so a stats file heavy in node_modules no longer trips the threshold. intelliStory.test.js grows 364 lines with cwd ≠ projectRoot fixtures (projectRoot: '/repo', invocationDir: '/repo/packages/ui') and a resolveConfigDirs() suite — the gap that let finding 1 through is now covered.
Review Table
| Priority | Category | Check | Status | Notes |
|---|---|---|---|---|
| High | Security | No hardcoded secrets or credentials | Pass | No credentials in the diff; auth flows through existing PercyClient token handling. |
| High | Security | Authentication/authorization checks present | N/A | CLI-side code; the new /intelli_story calls reuse existing token auth. |
| High | Security | Input validation and sanitization | Pass | assertSafeRef whitelists git refs before spawnSync; statsFile narrowed to a bare .json basename; spawnSync runs without a shell. User globs are now validated up front via patternError. Dedicated security lens is disabled in this orchestrator. |
| High | Security | No IDOR — resource ownership validated | N/A | Server-side concern; endpoints are keyed by the caller's own build id under its own token. |
| High | Security | No SQL injection (parameterized queries) | N/A | No SQL in this codebase. |
| High | Correctness | Logic is correct, handles edge cases | Pass | The path-basis defect is fixed, with both readings handled by direction of harm. Remaining gaps are Medium/Low. |
| High | Correctness | Error handling is explicit, no swallowed exceptions | Pass | Glob compile failures are no longer swallowed, and incomplete graphs now bail past a threshold. Finding 2 (unwrapped client calls) remains, at Medium. |
| High | Correctness | No race conditions or concurrency issues | Pass | pollGraphStatus is a bounded sequential loop; no shared mutable state beyond the per-client intelliStoryStats counter. |
| Medium | Testing | New code has corresponding tests | Pass | +364 test lines this push, covering resolvePattern, resolveConfigDirs, and the <rootDir>/ opt-out. |
| Medium | Testing | Error paths and edge cases tested | Pass | The cwd ≠ projectRoot gap is closed. Still untested: a rejecting client call (finding 2). |
| Medium | Testing | Existing tests still pass (no regressions) | Pass | 46 checks green, incl. Lint, Build & verify executable, Regression, all cli-command suites and the Node 20 leg. Test @percy/core still pending at review time — unrelated to the changed packages. |
| Medium | Performance | No N+1 queries or unbounded data fetching | Pass | One GET, one POST, bounded poll (12 × 5s). In-memory stats parsing. |
| Medium | Performance | Long-running tasks use background jobs | Pass | Graph generation is enqueued server-side and polled. |
| Medium | Quality | Follows existing codebase patterns | Pass | Matches PercyClient conventions, @percy/logger scoping, and the package's exports subpath pattern. |
| Medium | Quality | Changes are focused (single concern) | Pass | IntelliStory plus the binary-crash fix it depends on and the CI leg that exercises the lockfile path. |
| Low | Quality | Meaningful names, no dead code | Pass | stream-json confirmed absent. New helpers (resolvePattern, patternError, rootReadingHit, suggestRewrite) are well named and each has a stated reason to exist. |
| Low | Quality | Comments explain why, not what | Pass | The new comments are unusually good — they record the direction of harm reasoning behind each asymmetry ("recognising one directory too many only costs snapshots, while missing the real one loses comparisons"), which is exactly what a future maintainer needs. |
| Low | Quality | No unnecessary dependencies added | Pass | glob-to-regexp used; snyk-nodejs-lockfile-parser correctly optional with a Node 20 CI leg. |
Findings
2. API/network failures still escape as raw errors instead of IntelliStoryBailError
- File:
packages/cli-command/src/intelliStory.js:292,:331,:555 - Severity: Medium
- Reviewer: stack:code-reviewer (reported independently by both prior passes); re-verified unfixed at this head
- Issue: Verified still open:
getStatus('intelli_story_graph', …)inpollGraphStatus(:292),getIntelliStorySnapshotNameToCommit(:331), andgenerateIntelliStoryGraph(:555) are all bareawaits with no surrounding try/catch. Every other failure mode in this file — git, fs, config, lockfile, malformed glob, degraded graph — is deliberately converted toIntelliStoryBailErrorso the caller can degrade to a full snapshot run. A transient 5xx, timeout, or connection reset from these three still propagates as a plainError, so if@percy/storybookcatchesIntelliStoryBailErrorspecifically, an API blip hard-fails the storybook command instead of falling back. The graph-job timeout path is handled correctly (:561); only thrown exceptions leak. Not a merge blocker under the verdict rule, but it is now the last inconsistency in an otherwise uniformly fail-safe module. - Suggestion:
Same for
let baseLookup; try { baseLookup = await percy.client.getIntelliStorySnapshotNameToCommit(percy.build?.id); } catch (e) { throw new IntelliStoryBailError(`IntelliStory: base build lookup failed: ${e.message}; running full snapshot set`); }
generateIntelliStoryGraph; inpollGraphStatus, treat a throw as a non-donestatus so the existing:561bail covers it. Add tests where each client method rejects.
6. Bare-extension globs still don't match nested paths, and the contract is undocumented
- File:
packages/cli-command/src/intelliStory.js:28 - Severity: Medium
- Reviewer: stack:code-reviewer (first pass)
- Issue:
glob-to-regexp('*.css', { globstar: true })compiles to^([^/]*)\.css$, sobailOnChanges: ['*.css']never matchessrc/Button.css. This is standard glob behavior (minimatch, picomatch, GitHub Actionspaths:all agree) and is deliberately asserted by the tests, so it is not a defect. It is still listed because the newrootReadingHitwarning covers the basis mistake but not the depth mistake — a user who writes*.cssgets no warning at all — and there remains no documentation forbailOnChanges/untraced/configDiranywhere in the repo. - Suggestion: Document the semantics now that the basis rules are non-trivial (invocation-relative,
<rootDir>/opt-out, path-segment-scoped globs). Optionally extend the existing warning: if a pattern has no/and no**, and some affected node matches**/<pattern>, warn the same way the root-reading case does.
7–11. Carried-over Lows, unchanged at this head
noRequireBinding.test.js:40—FORBIDDENstill only matches a single-lineconst/let/var require = createRequire. The same crash returns viaimport { createRequire as require } from 'module'or a line-wrapped declaration. Broaden to flag any binding of the barerequireidentifier.intelliStory.js:229—mapEntrystill gates normalization oncopy.type === 'src'. Drop the gate;resolveAndIndexalready passes non-absolute values through untouched.intelliStory.js:138—gitDiffNamesstill omits--no-renameswhilegetAffectedFileLocationsuses it, so the two views of one diff disagree for renamed files.intelliStory.jsresolveAndIndex—path.relativedoes not case-fold, so a casing divergence between bundler module ids andgit rev-parse --show-toplevelyields..-prefixed index keys that never match. Mechanism is real, real-world trigger still unconfirmed. Cheapest guard: warn when the computed relative path starts with...graphTraceTemplate.html:8,:12—trace.htmlstill loads Drawflow fromcdn.jsdelivr.net, so the diagnostic renders blank in an air-gapped or locked-down CI runner. Vendor the bundle.
12. Bail visibility — partially addressed
- Severity: Low
- Issue: This push adds ten
log.warncalls, so pattern andconfigDirproblems are now visible. TheIntelliStoryBailErrorthrow sites themselves still don't log before throwing, so whether a user learns that filtering did not apply continues to depend on the@percy/storybookcatch site — outside this repo and still a follow-up per the PR body. - Suggestion: A small
bail(message, log)helper that warns and throws.
Informational
devDependenciesexcluded from the lockfile gate (lockfileDiff.js:53-63, tested): Storybook addons are usually devDependencies imported directly bypreview.js/story files, so a pure addon bump with no source diff won't surface here. Explicit and tested, so presumably intentional — worth a sanity check with the team..semgrepignorestill suppressesintelliStory.jsfile-wide rather than at the flagged joins. The file grew ~290 lines this push, which widens the unscanned surface; the documented rationale (inline// nosemgrepunsupported by the CI semgrep version) still holds. Worth narrowing when tooling allows.- Version strings moved
1.32.6-beta.3→1.32.6via the master release merge; no functional change.
Verdict: PASS
What
Adds IntelliStory: for a Storybook build, it diffs against a baseline (explicit or API-predicted) and filters the snapshot set down to only the stories a change actually affects, bailing to a full snapshot run whenever it can't safely reason about the change.
This restores the functionality previously reverted in #2310, under the IntelliStory name.
cli-command
src/intelliStory.js—applyIntelliStory()andIntelliStoryBailError, exported via the new./intelliStorysubpath.src/lockfileDiff.js,src/graphTrace.js(+graphTraceTemplate.html).glob-to-regexp,stream-json, and optionalsnyk-nodejs-lockfile-parserdependencies.client
getStatus()accepts theintelli_story_graphjob type (sync response carries the graph payload).getIntelliStorySnapshotNameToCommit()andgenerateIntelliStoryGraph()hit the/intelli_storyendpoints.Binary-crash fix (folds in #2306)
intelliStory.jsandlockfileDiff.jsbindcreateRequiretocjsRequire(notrequire). When transpiled to CommonJS for the packaged binary, naming itrequirecollides with Babel's preset-env + transform-import-meta and crashes on startup withTypeError: _require is not a function. A new static regression guard,test/noRequireBinding.test.js, scans allpackages/*/srcfor the footgun (with a matching.semgrepignorerationale).Testing
proxy.test.jsfailures are unrelated — a Node 22Invalid URLmessage-format difference in files this PR does not touch.)yarn buildcompiles cleanly;dist/intelliStory.jsemitscjsRequire.Follow-up
The external
@percy/storybookconsumer will need the matching@percy/cli-command/intelliStory/applyIntelliStoryupdate.