Skip to content

PPLT-5844: add IntelliStory storybook affected-story filtering - #2339

Open
RaghavsBrowserStack wants to merge 23 commits into
masterfrom
PPLT-5844
Open

PPLT-5844: add IntelliStory storybook affected-story filtering#2339
RaghavsBrowserStack wants to merge 23 commits into
masterfrom
PPLT-5844

Conversation

@RaghavsBrowserStack

Copy link
Copy Markdown
Contributor

What

Adds IntelliStory: for a Storybook build, it diffs against a baseline (explicit or API-predicted) and filters the snapshot set down to only the stories a change actually affects, bailing to a full snapshot run whenever it can't safely reason about the change.

This restores the functionality previously reverted in #2310, under the IntelliStory name.

cli-command

  • src/intelliStory.jsapplyIntelliStory() and IntelliStoryBailError, exported via the new ./intelliStory subpath.
  • src/lockfileDiff.js, src/graphTrace.js (+ graphTraceTemplate.html).
  • Adds glob-to-regexp, stream-json, and optional snyk-nodejs-lockfile-parser dependencies.

client

  • getStatus() accepts the intelli_story_graph job type (sync response carries the graph payload).
  • getIntelliStorySnapshotNameToCommit() and generateIntelliStoryGraph() hit the /intelli_story endpoints.

Binary-crash fix (folds in #2306)

intelliStory.js and lockfileDiff.js bind createRequire to cjsRequire (not require). When transpiled to CommonJS for the packaged binary, naming it require collides with Babel's preset-env + transform-import-meta and crashes on startup with TypeError: _require is not a function. A new static regression guard, test/noRequireBinding.test.js, scans all packages/*/src for the footgun (with a matching .semgrepignore rationale).

Testing

  • cli-command: 145/145 specs pass (intelliStory, lockfileDiff, graphTrace, index, noRequireBinding).
  • client: IntelliStory client tests pass. (Two pre-existing proxy.test.js failures are unrelated — a Node 22 Invalid URL message-format difference in files this PR does not touch.)
  • yarn build compiles cleanly; dist/intelliStory.js emits cjsRequire.

Follow-up

The external @percy/storybook consumer will need the matching @percy/cli-command/intelliStory / applyIntelliStory update.

…iltering

Introduces IntelliStory: given a Storybook build, it diffs against a baseline
(explicit or API-predicted) and filters the snapshot set down to the stories a
change actually affects, bailing to a full run whenever it can't reason about
the change.

- cli-command: intelliStory.js (applyIntelliStory, IntelliStoryBailError),
  lockfileDiff.js and graphTrace.js, exported via the ./intelliStory subpath;
  adds glob-to-regexp, stream-json and optional snyk-nodejs-lockfile-parser.
- client: getStatus() accepts the intelli_story_graph job type, plus
  getIntelliStorySnapshotNameToCommit() and generateIntelliStoryGraph()
  hitting the /intelli_story endpoints.

Binds createRequire to cjsRequire (not `require`) in intelliStory.js and
lockfileDiff.js so the CommonJS-transpiled packaged binary doesn't crash with
"_require is not a function", and adds noRequireBinding.test.js as a static
regression guard (with the matching .semgrepignore rationale).
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
// applies, without exporting the module-private helpers it delegates to.
function embeddedJson(html, name) {
// test-only helper; name is a hardcoded literal ('vertices'/'edges') from the test, not external input (reviewed, approved by security)
let match = html.match(new RegExp(`const ${name} = (.*);`)); // nosemgrep
…trim comments

- intelliStory.js: read the enriched stats file with a single
  JSON.parse(fs.readFileSync(...)) instead of streaming it with stream-json.
  The stats payload is only a few MB, so in-memory parsing is simpler and
  removes the stream-json dependency (and, with it, the last createRequire
  binding in this file — the packaged-binary footgun now only exists in
  lockfileDiff.js for the optional snyk parser).
- Remove stream-json from cli-command dependencies and the lockfile.
- Strip explanatory comments introduced with the feature across the
  IntelliStory source, tests and trace template. Coverage directives
  (istanbul ignore), semgrep suppressions (nosemgrep) and the cjsRequire /
  loadSnyk rationale in lockfileDiff.js are kept — removing them would break
  the 100% coverage gate and the semgrep scan.
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
@RaghavsBrowserStack RaghavsBrowserStack changed the title feat(cli-command,client): add IntelliStory storybook affected-story filtering PPLT-5844: add IntelliStory storybook affected-story filtering Jul 15, 2026
@RaghavsBrowserStack
RaghavsBrowserStack marked this pull request as ready for review July 15, 2026 14:29
@RaghavsBrowserStack
RaghavsBrowserStack requested a review from a team as a code owner July 15, 2026 14:29
Comment thread packages/cli-command/src/intelliStory.js Fixed
Comment thread packages/cli-command/src/intelliStory.js Fixed
RaghavsBrowserStack and others added 3 commits July 21, 2026 13:20
Add tests for the graceful-bail paths added in "apply suggested fixed":
- malformed stats-file JSON (validateAndReadStats)
- unreadable current lockfile (getAffectedPackages)
- unreadable package.json (getAffectedPackages)

These run on Node 14 (they bail before the Node>=18 snyk path), fixing the
lines/statements coverage-threshold failure on the Node 14 test job.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GLOB_CHARS only detected * and ? so brace ({a,b}) and bracket ([abc])
patterns fell through to exact-match, silently disabling the intended
glob semantics for these documented config options.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@RaghavsBrowserStack

Copy link
Copy Markdown
Contributor Author

Claude Code PR Review

PR: #2339Head: 001c036Reviewers: stack:code-reviewer

Summary

Adds IntelliStory — an affected-story graph feature for @percy/storybook: parses Storybook build stats, diffs against a git baseline (incl. lockfile/dependency diffing via optional snyk-nodejs-lockfile-parser), enqueues a server-side affected-story graph job, tags snapshots for selection, and optionally renders a debug trace.html. Plumbs intelliStory/storybookPath through @percy/client and @percy/core, and adds a Node 20 CI leg for the Node-18+-only snyk path.

Review Table

Priority Category Check Status Notes
High Security No hardcoded secrets or credentials Pass None introduced.
High Security Authentication/authorization checks present N/A No auth surface changed.
High Security Input validation and sanitization Pass assertSafeRef blocks flag-injection; safeJson escapes HTML/JS breakouts; paths anchored via basename.
High Security No IDOR — resource ownership validated N/A No resource-ownership surface.
High Security No SQL injection (parameterized queries) N/A No SQL.
High Correctness Logic is correct, handles edge cases Fail Finding 1 ($-pattern .replace) still open. Finding 2 (glob detection gap) resolved in 001c036.
High Correctness Error handling is explicit, no swallowed exceptions Pass* Mostly graceful IntelliStoryBailError; one inconsistent rethrow — Finding 3.
High Correctness No race conditions or concurrency issues Pass No new concurrency.
Medium Testing New code has corresponding tests Pass Broad coverage across new modules.
Medium Testing Error paths and edge cases tested Pass* Package-affected merge branch istanbul ignored rather than tested — Finding 5.
Medium Testing Existing tests still pass (no regressions) Pass Not re-run here; CI covers.
Medium Performance No N+1 queries or unbounded data fetching N/A No queries.
Medium Performance Long-running tasks use background jobs Pass Graph job is server-side/async with timeout bail.
Medium Quality Follows existing codebase patterns Pass Consistent with client/core plumbing conventions.
Medium Quality Changes are focused (single concern) Pass Scoped to IntelliStory feature.
Low Quality Meaningful names, no dead code Pass Clear naming.
Low Quality Comments explain why, not what Pass .semgrepignore rationale well-documented.
Low Quality No unnecessary dependencies added Pass snyk-nodejs-lockfile-parser is optional/lazy-loaded.

Findings

  • File: packages/cli-command/src/graphTrace.js:114-116

  • Severity: Medium

  • Reviewer: stack:code-reviewer (verified)

  • Issue: .replace('__VERTICES_JSON__', safeJson(...)) uses plain-string replacement values. JS honors $$, $&, $`, $', $n special patterns in the replacement string even when the search arg is a literal. Embedded JSON contains arbitrary file paths; a path with $' re-injects the rest of the template, breaking the embedded JS. safeJson's escaping doesn't cover this — the corruption happens in the .replace call itself.

  • Suggestion: Pass a function as the replacement: .replace('__VERTICES_JSON__', () => safeJson(...)). Add a graphTrace.test.js case with $&/$'/$$ in a vertex name.

  • File: packages/cli-command/src/intelliStory.js:16, 34-41

  • Severity: Medium — ✅ RESOLVED in 001c036

  • Reviewer: stack:code-reviewer (verified)

  • Issue: GLOB_CHARS = /[*?]/ only routed */? patterns through the glob matcher, but patternToRegex enables {extended:true, globstar:true} (brace/bracket support). A configured bailOnChanges: ['src/{a,b}.js'] or bracket pattern silently fell through to literal str === pattern and never matched — silently disabling a documented config option (bailOnChanges/untraced).

  • Fix applied: Broadened detection to const GLOB_CHARS = /[*?{}[\]]/;. Added brace- and bracket-glob tests to assertNoBailOnChanges() and enforceUntraced() — verified failing under the old regex and passing under the fix (full 156-spec suite green).

  • File: packages/cli-command/src/intelliStory.js:326-336

  • Severity: Medium

  • Reviewer: stack:code-reviewer

  • Issue: Only SNYK_LOCKFILE_PARSER_UNAVAILABLE is converted to a graceful IntelliStoryBailError; a buildDepTree parse failure (malformed/older-ref lockfile) rethrows as a raw Error. Every other failure path degrades to "running full snapshot set."

  • Suggestion: Confirm the @percy/storybook caller treats all applyIntelliStory throws as fall-back-to-full-set. If not, wrap this rethrow in IntelliStoryBailError for consistency.

  • File: packages/cli-command/src/intelliStory.js:67-70

  • Severity: Low

  • Reviewer: stack:code-reviewer

  • Issue: assertSafeRef regex correctly blocks leading -, but also rejects ordinary safe ref syntax like HEAD~5 / abc123^2.

  • Suggestion: Confirm intentional, or widen charset to include ~^ (still safe given the leading-char restriction).

  • File: packages/cli-command/src/intelliStory.js:438-440

  • Severity: Low

  • Reviewer: stack:code-reviewer

  • Issue: The packageAffectedNodes merge into affectedNodes (the actual mechanism folding lockfile-driven dep changes into the graph) is /* istanbul ignore next */ rather than covered by a real test.

  • Suggestion: Add an itPosix-style Node≥18 integration test exercising the merge end-to-end.

  • File: packages/cli-command/src/intelliStory.js (resolveAndIndex)

  • Severity: Low

  • Reviewer: stack:code-reviewer

  • Issue: Absolute module ids resolving outside projectRoot (symlinked pkg, monorepo sibling) produce ../-prefixed paths added straight into the files index without a guard.

  • Suggestion: Bail or drop entries where rel.startsWith('..').

  • File: .semgrepignore:118-131 / packages/client/src/client.js:2591-2592 / .github/workflows/test.yml

  • Severity: Info

  • Reviewer: stack:code-reviewer

  • Issue: (a) Whole-file semgrep suppression means future traversal code in the growing intelliStory.js goes unchecked. (b) client.intelliStoryStats is lazily-created, never reset — accumulates across builds if a client is reused. (c) New test-node20 CI job (workflow change) — flagged for visibility; mirror logic looks consistent, no action requested.

  • Suggestion: Consider narrowing semgrep scope over time; reset stats in createBuild.


Verdict: PASS

@RaghavsBrowserStack

Copy link
Copy Markdown
Contributor Author

Claude Code PR Review

PR: #2339Head: 966c05dReviewers: stack:code-reviewer (definition run inline — the colon-prefixed agent name is not registered as a launchable agent type), findings then re-verified against source by the orchestrator

Summary

Adds IntelliStory to @percy/cli-command: for a Storybook build it resolves a baseline (explicit or API-predicted), diffs the working tree against it, ships the enriched module graph to the new /intelli_story endpoints, and tags each snapshot with intelliStory/storybookPath so the API performs affected-story selection server-side — bailing to a full snapshot run via IntelliStoryBailError whenever it can't reason safely about a change. Also folds in the packaged-binary crash fix (createRequire bound as cjsRequire, guarded by a new static noRequireBinding test) and a Node 20 CI leg for the Snyk-backed lockfile-diff path.

The bail-first posture is the right architecture for a feature that can silently drop visual coverage, and it is applied thoroughly — 15+ explicit bail sites covering missing build id, unreadable/invalid stats, git failures, ambiguous or absent lockfiles, manifest changes spanning directories, browser-upgrade builds, and graph-job timeouts. All four Medium findings below are gaps in that otherwise-sound safety net, not architectural problems.

Review Table

Priority Category Check Status Notes
High Security No hardcoded secrets or credentials Pass No credentials in the diff; auth continues to flow through the existing PercyClient token handling.
High Security Authentication/authorization checks present N/A CLI-side code; the new /intelli_story calls reuse PercyClient's existing token auth.
High Security Input validation and sanitization Pass assertSafeRef whitelists git refs before they reach spawnSync (intelliStory.js:65); statsFile is narrowed to a bare .json basename (:192); spawnSync is used without a shell. The dedicated security lens is disabled in this orchestrator, so this row is a correctness-level read only.
High Security No IDOR — resource ownership validated N/A Server-side concern; the new endpoints are keyed by the caller's own build id under its own token.
High Security No SQL injection (parameterized queries) N/A No SQL in this codebase.
High Correctness Logic is correct, handles edge cases Pass Extensive and largely correct bail coverage. Four Medium gaps: findings 1–4.
High Correctness Error handling is explicit, no swallowed exceptions Pass Every git/fs/config failure is wrapped in IntelliStoryBailError. Gaps: the three API calls are not (finding 1), and dropped stats modules are logged at debug only (finding 4).
High Correctness No race conditions or concurrency issues Pass pollGraphStatus is a bounded sequential loop (12 × 5s); no shared mutable state beyond the per-client intelliStoryStats counter.
Medium Testing New code has corresponding tests Pass ~1200 new test lines across intelliStory, lockfileDiff, graphTrace, index, noRequireBinding; PR reports 145/145 cli-command specs green.
Medium Testing Error paths and edge cases tested Pass Bail branches are covered individually (commit e071c952 restored 100% coverage). The glob semantics in finding 3 are asserted deliberately by intelliStory.test.js:286-289.
Medium Testing Existing tests still pass (no regressions) Pass All CI checks green on 966c05d: Lint, Build, Build & verify executable, Regression, every package suite (incl. the new Node 20 leg), CodeQL, Semgrep OSS.
Medium Performance No N+1 queries or unbounded data fetching Pass One snapshot-name-to-commit GET, one generate-graph POST, and a bounded poll (max 12 attempts / ~60s). Stats parsing is in-memory after stream-json was dropped.
Medium Performance Long-running tasks use background jobs Pass Graph generation is enqueued server-side and polled; selection happens server-side at snapshot-post time.
Medium Quality Follows existing codebase patterns Pass Matches PercyClient method/identifier conventions, @percy/logger scoping, and the package's exports subpath pattern.
Medium Quality Changes are focused (single concern) Pass IntelliStory plus the binary-crash fix it depends on (folds in #2306) and the CI leg needed to exercise the lockfile path — all justified in the PR body.
Low Quality Meaningful names, no dead code Pass stream-json confirmed fully removed from packages/cli-command/package.json and unreferenced — not a leftover. Minor: findings 5–7.
Low Quality Comments explain why, not what Pass Notably good: the cjsRequire rationale, the server-side-selection comment at intelliStory.js:437-447, and the .semgrepignore entries all explain why.
Low Quality No unnecessary dependencies added Pass glob-to-regexp is small and used; snyk-nodejs-lockfile-parser is correctly an optionalDependency with a Node 20 CI leg to cover it.

Findings

1. API/network failures escape as raw errors instead of IntelliStoryBailError

  • File: packages/cli-command/src/intelliStory.js:181, :220, :354
  • Severity: Medium
  • Reviewer: stack:code-reviewer (refined by orchestrator verification)
  • Issue: Every git, fs, config, and lockfile failure in this module is wrapped in IntelliStoryBailError so the caller can degrade to a full snapshot run. The three percy.client calls — getIntelliStorySnapshotNameToCommit (:220), getStatus('intelli_story_graph', …) inside pollGraphStatus (:181), and generateIntelliStoryGraph (:354) — are not. A transient 5xx, timeout, or DNS failure therefore propagates as a plain Error. If @percy/storybook catches IntelliStoryBailError specifically (the type this module exports for exactly that purpose), an API blip hard-fails the whole storybook command instead of falling back to a full run — the opposite of the module's fail-safe posture. The graph-job timeout path is handled correctly (:359); it is only thrown exceptions that leak.
  • Suggestion: Wrap each client call and rethrow, e.g.
    let baseLookup;
    try {
      baseLookup = await percy.client.getIntelliStorySnapshotNameToCommit(percy.build?.id);
    } catch (e) {
      throw new IntelliStoryBailError(`IntelliStory: base build lookup failed: ${e.message}; running full snapshot set`);
    }
    Same for generateIntelliStoryGraph, and in pollGraphStatus treat a throw as a non-done status.

2. .storybook config directory is hardcoded — a custom configDir never trips the bail

  • File: packages/cli-command/src/intelliStory.js:114, :244
  • Severity: Medium
  • Reviewer: stack:code-reviewer (verified)
  • Issue: Both the graph exclusion (EXCLUDED_DIRS = new Set(['node_modules', '.storybook'])) and the bail check (assertNoDotStorybookChange) match the literal path segment .storybook. Storybook's config directory is configurable (-c <dir> / configDir), and intelliStoryConfig has no field to communicate it. For a project using e.g. config/storybook/, a change to main.js or preview.js — which can affect every story — produces no bail, and filtering proceeds as though nothing global changed. This is the under-snapshot direction: real visual regressions ship unnoticed.
  • Suggestion: Thread the resolved config-dir name through intelliStoryConfig from @percy/storybook into both call sites, defaulting to .storybook:
    export function assertNoDotStorybookChange(affectedNodes, configDir = '.storybook') {
      const hit = affectedNodes.find(p => p.split(/[/\\]/).includes(configDir));
      
    }

3. Bare-extension bailOnChanges globs silently match nothing below the repo root

  • File: packages/cli-command/src/intelliStory.js:28 (patternToRegex), consumed at :251
  • Severity: Medium
  • Reviewer: stack:code-reviewer (severity downgraded from High by the orchestrator — see below)
  • Issue: With { extended: true, globstar: true }, glob-to-regexp('*.css') compiles to ^([^/]*)\.css$ — verified locally — so it matches only root-level files, never src/components/Button.css. A user who configures bailOnChanges: ['*.css'] expecting "bail on any stylesheet change" gets no bail, and IntelliStory filters as if nothing unsafe changed. untraced shares the semantics but fails safe (nothing is dropped, so more snapshots are taken).
    Downgraded from the sub-reviewer's High because this is standard glob behaviorminimatch, picomatch, and GitHub Actions paths: filters all treat a bare *.ext as root-only — and the tests assert it deliberately (intelliStory.test.js:266, :286-289), so it is a considered choice rather than a coding error. Still worth flagging: bailOnChanges is the one user-facing safety valve and the failure mode is silent.
  • Suggestion: Either document that patterns are path-segment-scoped and **/*.css is required (there is currently no documentation for bailOnChanges/untraced anywhere in the repo), or auto-broaden bare-name patterns before compiling: const p = pattern.includes('/') ? pattern : '**/' + pattern;. If the current semantics stay, rename the test to state the intent ('a bare-extension pattern does not match nested paths') so the behavior reads as deliberate.

4. Stats modules dropped from the graph are logged only at debug, with no threshold

  • File: packages/cli-command/src/intelliStory.js:132-133, :161-166
  • Severity: Medium
  • Reviewer: orchestrator
  • Issue: transformModule returns null for any module whose id does not resolve to an absolute path, and readStats counts those into droppedModules and emits a single log.debug. Missing modules mean missing edges in the dependency graph, and a missing edge is exactly what makes the API judge a story unaffected. If a Storybook/webpack version emits a large share of synthetic or virtual module ids, the graph is quietly degraded and the run still proceeds — visible only under --verbose. Every other incomplete-data condition in this file bails.
  • Suggestion: Escalate the existing message to log.warn, and bail above a threshold:
    if (droppedModules) {
      const ratio = droppedModules / rawModules.length;
      if (ratio > 0.1) throw new IntelliStoryBailError(`IntelliStory: ${droppedModules}/${rawModules.length} modules had unresolved ids; graph would be incomplete; running full snapshot set`);
      log.warn(`IntelliStory: dropped ${droppedModules} module(s) with unresolved (non-absolute) ids`);
    }

5. mapEntry normalizes source only for type === 'src'

  • File: packages/cli-command/src/intelliStory.js:137
  • Severity: Low
  • Reviewer: stack:code-reviewer
  • Issue: resolveAndIndex runs only when copy.type === 'src'. Fixtures only exercise 'src' and 'module' (the latter a no-op, since package specifiers are never absolute). If the external enrichment tool ever emits a third type carrying an absolute path — an asset, CSS, or JSON import — that raw absolute path ships unnormalized in the modules payload: it leaks the developer's home or CI runner path, and it cannot be correlated against the project-relative affectedNodes, which is a false-negative path.
  • Suggestion: Drop the type gate — resolveAndIndex already returns non-absolute values untouched, so running it for every string source is safe and type-agnostic.

6. gitDiffNames omits --no-renames while getAffectedFileLocations uses it

  • File: packages/cli-command/src/intelliStory.js:75 vs :84
  • Severity: Low
  • Reviewer: stack:code-reviewer (correction: rename detection is git's default since 2.9, so the inconsistency is present out of the box, not only under custom config)
  • Issue: affectedNodes comes from git diff --name-only with rename detection active, which collapses a rename to the new path alone; getAffectedFileLocations runs the same diff with --no-renames, which yields a delete + add pair. The two views of one diff can disagree for renamed files. Low practical impact — the old path no longer exists to be imported — but it makes the two graph inputs non-comparable.
  • Suggestion: Add --no-renames to the gitDiffNames invocation for determinism and parity with :84.

7. trace.html loads Drawflow from a public CDN

  • File: packages/cli-command/src/graphTraceTemplate.html:8, :12
  • Severity: Low
  • Reviewer: stack:code-reviewer (verified)
  • Issue: The debug trace viewer pulls Drawflow's CSS and JS from cdn.jsdelivr.net at view time. This is opt-in (trace: true) and off the production path, but it is precisely the artifact someone opens after a bail or a mis-selection — often inside a locked-down or air-gapped CI environment where the CDN is unreachable and the trace renders blank.
  • Suggestion: Vendor the (small) Drawflow bundle into the package and inline it into the template.

8. Whole-file .semgrepignore suppression for intelliStory.js

  • File: .semgrepignore:49-57
  • Severity: Low
  • Reviewer: stack:code-reviewer
  • Issue: The suppression is file-level rather than scoped to the specific path.join/path.resolve sites that trigger the false positive. intelliStory.js does a fair amount of fs/path work, so any genuinely unsafe join added later goes unscanned. The stated rationale (inline // nosemgrep not honored by the CI semgrep version) is legitimate and well documented.
  • Suggestion: Track a follow-up to narrow the suppression once the CI semgrep version honors inline annotations.

9. No bail is logged at the throw site

  • File: packages/cli-command/src/intelliStory.js — all IntelliStoryBailError throw sites
  • Severity: Low
  • Reviewer: stack:code-reviewer
  • Issue: Every bail constructs a carefully worded message but none logs at warn/info before throwing, so whether a user ever learns that filtering did not apply depends entirely on the catch site in @percy/storybook — outside this repo and, per the PR body, still a follow-up. If that catcher logs at debug or swallows while falling back, the fail-safe is invisible.
  • Suggestion: log.warn(message) at the throw site (or inside a small bail(message, log) helper) so visibility does not depend on the external consumer.

Verified non-issues

  • stream-json leftover: confirmed absent from packages/cli-command/package.json and unreferenced anywhere in cli-command — correctly removed in 9c525f10.
  • Backward compatibility: the new snapshotSchema fields (intelliStory, storybookPath) are optional, and createSnapshot sends 'intelli-story': intelliStory || null / 'storybook-path': storybookPath || null, so non-IntelliStory callers are unchanged — cli-upload/test/upload.test.js pins exactly that.
  • getStatus response shape: intelli_story_graph returns a top-level { status, data }, unlike the batched-by-id shape used for snapshot/comparison. pollGraphStatus and client.test.js:890-905 agree on it; an API-contract quirk, not a defect.

Verdict: PASS

@RaghavsBrowserStack

Copy link
Copy Markdown
Contributor Author

Claude Code PR Review

PR: #2339Head: 01aa1f6Reviewers: stack:code-reviewer (definition run inline — the colon-prefixed agent name is not registered as a launchable agent type), second independent pass; all findings re-verified against source by the orchestrator

Summary

Adds IntelliStory to @percy/cli-command: for a Storybook build it resolves a baseline (explicit or API-predicted), diffs the working tree against it, ships the enriched module graph to the new /intelli_story endpoints, and tags each snapshot with intelliStory/storybookPath so the API performs affected-story selection server-side — bailing to a full snapshot run via IntelliStoryBailError whenever it can't reason safely about a change. Also folds in the packaged-binary crash fix (createRequire bound as cjsRequire, guarded by a static noRequireBinding test) and a Node 20 CI leg for the Snyk-backed lockfile-diff path.

Note on this run. Head moved from 966c05d to 01aa1f6, but both new commits (01aa1f6, f6ef573) are Merge branch 'master' into PPLT-5844git diff 966c05d 01aa1f6 over the PR-owned paths is empty, so the reviewable code is byte-identical to the previously reviewed head. The fast-path did not fire because the intermediate merge f6ef573 carries no prior Claude Code Review status, so a full review ran. This second, independent pass surfaced finding 1, which the first pass missed. The code has not regressed — the assessment has sharpened, and the verdict changes accordingly.

Review Table

Priority Category Check Status Notes
High Security No hardcoded secrets or credentials Pass No credentials in the diff; auth continues to flow through existing PercyClient token handling.
High Security Authentication/authorization checks present N/A CLI-side code; the new /intelli_story calls reuse PercyClient's existing token auth.
High Security Input validation and sanitization Pass assertSafeRef whitelists git refs before they reach spawnSync (intelliStory.js:65); statsFile is narrowed to a bare .json basename (:192); spawnSync runs without a shell. The dedicated security lens is disabled in this orchestrator, so this row is a correctness-level read only.
High Security No IDOR — resource ownership validated N/A Server-side concern; the new endpoints are keyed by the caller's own build id under its own token.
High Security No SQL injection (parameterized queries) N/A No SQL in this codebase.
High Correctness Logic is correct, handles edge cases Fail Finding 1: bailOnChanges/untraced globs are matched against repo-root-relative paths with no rebasing, while story importPaths in the same function are rebased from the invocation directory — so the user's safety valve is silently inert for monorepo/subdirectory invocation.
High Correctness Error handling is explicit, no swallowed exceptions Pass Every git/fs/config failure is wrapped in IntelliStoryBailError. Gaps are Medium, not High: the three API calls are unwrapped (finding 2), malformed globs are swallowed to false (finding 4), and dropped stats modules are logged at debug only (finding 5).
High Correctness No race conditions or concurrency issues Pass pollGraphStatus is a bounded sequential loop (12 × 5s); no shared mutable state beyond the per-client intelliStoryStats counter.
Medium Testing New code has corresponding tests Pass ~1200 new test lines across intelliStory, lockfileDiff, graphTrace, index, noRequireBinding; PR reports 145/145 cli-command specs green.
Medium Testing Error paths and edge cases tested Fail Bail branches are covered individually, but every fixture builds a git repo with files at its root — there is no cwd ≠ projectRoot case, which is exactly why finding 1 went unnoticed. No test simulates a rejecting client call (finding 2).
Medium Testing Existing tests still pass (no regressions) Pass All CI checks were green on 966c05d, whose PR-owned tree is identical to this head.
Medium Performance No N+1 queries or unbounded data fetching Pass One snapshot-name-to-commit GET, one generate-graph POST, and a bounded poll (max 12 attempts / ~60s). Stats parsing is in-memory after stream-json was dropped.
Medium Performance Long-running tasks use background jobs Pass Graph generation is enqueued server-side and polled; selection happens server-side at snapshot-post time.
Medium Quality Follows existing codebase patterns Pass Matches PercyClient method/identifier conventions, @percy/logger scoping, and the package's exports subpath pattern.
Medium Quality Changes are focused (single concern) Pass IntelliStory plus the binary-crash fix it depends on (folds in #2306) and the CI leg needed to exercise the lockfile path — all justified in the PR body.
Low Quality Meaningful names, no dead code Pass stream-json confirmed fully removed from packages/cli-command/package.json and unreferenced — not a leftover.
Low Quality Comments explain why, not what Pass Notably good: the cjsRequire rationale, the server-side-selection comment at intelliStory.js:437-447, and the .semgrepignore entries all explain why.
Low Quality No unnecessary dependencies added Pass glob-to-regexp is small and used; snyk-nodejs-lockfile-parser is correctly an optionalDependency with a Node 20 CI leg to cover it.

Findings

1. bailOnChanges/untraced globs are never rebased to the invocation directory — the safety valve is silently inert in a monorepo

  • File: packages/cli-command/src/intelliStory.js:250-252, :259-262 (compare :415-426)

  • Severity: High

  • Reviewer: stack:code-reviewer (second pass); mechanism verified by the orchestrator

  • Issue: affectedNodes comes from git diff --name-only, which always emits repo-root-relative paths regardless of cwd. assertNoBailOnChanges and enforceUntraced match user-supplied globs directly against those paths, and glob-to-regexp fully anchors the result (^…$). Meanwhile, twenty lines further down, the same function builds invocationDir = process.cwd() and rebases each story's importPath from the invocation directory to project-relative (:415-426) — so the file demonstrably does expect cwd ≠ projectRoot. User globs get no such treatment.

    Concrete failure: a monorepo user runs cd packages/ui && percy storybook ./storybook-static with bailOnChanges: ['webpack.config.js']. A change to that file lands in affectedNodes as packages/ui/webpack.config.js; the pattern compiles to /^webpack\.config\.js$/ and does not match. No bail fires, no warning is logged, and IntelliStory filters as though nothing risky changed — the under-snapshot direction, where a genuine visual regression ships unreviewed. Monorepo-with-subdirectory is the mainstream Storybook layout, and bailOnChanges exists precisely as the escape hatch for changes the graph can't reason about.

    Not covered by any test: every fixture in intelliStory.test.js constructs a repo with files at its root, so no cwd ≠ projectRoot case is exercised.

  • Suggestion: Rebase user patterns the same way story paths already are, before matching:

    const rebase = g => {
      if (path.isAbsolute(g)) return path.relative(projectRoot, g).split(path.sep).join('/');
      const abs = path.resolve(invocationDir, g);
      const rel = path.relative(projectRoot, abs);
      return rel.startsWith('..') ? g : rel.split(path.sep).join('/');
    };

    and pass bailOnChanges.map(rebase) / untraced.map(rebase) into the two functions. Add a test with cwd set to a nested package directory. If root-relative globs are the intended contract instead, that must be documented and a bail should fire when cwd !== projectRoot and patterns are configured — silently doing nothing is the one outcome to avoid.

2. API/network failures escape as raw errors instead of IntelliStoryBailError

  • File: packages/cli-command/src/intelliStory.js:181, :220, :354
  • Severity: Medium
  • Reviewer: stack:code-reviewer — independently reported by both review passes
  • Issue: Every git, fs, config, and lockfile failure in this module is wrapped in IntelliStoryBailError so the caller can degrade to a full snapshot run. The three percy.client calls — getIntelliStorySnapshotNameToCommit (:220), getStatus('intelli_story_graph', …) inside pollGraphStatus (:181), and generateIntelliStoryGraph (:354) — are not. A transient 5xx, timeout, or connection reset therefore propagates as a plain Error. If @percy/storybook catches IntelliStoryBailError specifically (the type this module exports for exactly that purpose), an API blip hard-fails the whole storybook command instead of falling back to a full run — the opposite of the module's fail-safe posture. The graph-job timeout path is handled correctly (:359); only thrown exceptions leak. diffLockfileDeps (:316) has the same shape: it remaps only SNYK_LOCKFILE_PARSER_UNAVAILABLE and re-throws anything else raw.
  • Suggestion: Wrap each client call and rethrow, e.g.
    let baseLookup;
    try {
      baseLookup = await percy.client.getIntelliStorySnapshotNameToCommit(percy.build?.id);
    } catch (e) {
      throw new IntelliStoryBailError(`IntelliStory: base build lookup failed: ${e.message}; running full snapshot set`);
    }
    Same for generateIntelliStoryGraph; in pollGraphStatus, treat a throw as a non-done status. Add tests where each client method rejects.

3. .storybook config directory is hardcoded — a custom configDir never trips the bail

  • File: packages/cli-command/src/intelliStory.js:114, :244
  • Severity: Medium
  • Reviewer: stack:code-reviewer (first pass); verified
  • Issue: Both the graph exclusion (EXCLUDED_DIRS = new Set(['node_modules', '.storybook'])) and the bail check (assertNoDotStorybookChange) match the literal path segment .storybook. Storybook's config directory is configurable (-c <dir> / configDir), and intelliStoryConfig has no field to communicate it. For a project using e.g. config/storybook/, a change to main.js or preview.js — which can affect every story — produces no bail. Same under-snapshot direction as finding 1, and the two compound for monorepo users.
  • Suggestion: Thread the resolved config-dir name through intelliStoryConfig from @percy/storybook into both call sites, defaulting to .storybook:
    export function assertNoDotStorybookChange(affectedNodes, configDir = '.storybook') {
      const hit = affectedNodes.find(p => p.split(/[/\\]/).includes(configDir));
      
    }

4. A malformed glob is silently treated as "matches nothing"

  • File: packages/cli-command/src/intelliStory.js:34-43 (matchesPattern)
  • Severity: Medium
  • Reviewer: stack:code-reviewer (second pass); verified — glob-to-regexp('src/[') throws Unterminated character class, 'src/{a' throws Unterminated group
  • Issue: try { … } catch { return false } means a typo'd pattern in a user's config compiles-fails and is then treated as never matching. For untraced that fails safe (the file stays traced). For bailOnChanges it fails open: the guardrail is silently disabled with nothing logged anywhere. The over-length guard at :22 has the same shape and is explicitly tested as non-throwing (intelliStory.test.js:269-271).
  • Suggestion: log.warn when a pattern fails to compile — a broken guardrail must be visible:
    } catch (e) {
      log.warn(`IntelliStory: ignoring invalid pattern "${pattern}": ${e.message}`);
      return false;
    }
    Better still, validate the patterns once at config-parse time rather than per-path.

5. Stats modules dropped from the graph are logged only at debug, with no threshold

  • File: packages/cli-command/src/intelliStory.js:132-133, :161-166
  • Severity: Medium
  • Reviewer: orchestrator (first pass)
  • Issue: transformModule returns null for any module whose id does not resolve to an absolute path, and readStats counts those into droppedModules and emits a single log.debug. Missing modules mean missing edges in the dependency graph, and a missing edge is exactly what makes the API judge a story unaffected. If a Storybook/webpack version emits a large share of synthetic or virtual module ids, the graph is quietly degraded and the run proceeds — visible only under --verbose. Every other incomplete-data condition in this file bails.
  • Suggestion: Escalate to log.warn and bail above a threshold:
    if (droppedModules) {
      const ratio = droppedModules / rawModules.length;
      if (ratio > 0.1) throw new IntelliStoryBailError(`IntelliStory: ${droppedModules}/${rawModules.length} modules had unresolved ids; graph would be incomplete; running full snapshot set`);
      log.warn(`IntelliStory: dropped ${droppedModules} module(s) with unresolved (non-absolute) ids`);
    }

6. Bare-extension globs don't match nested paths

  • File: packages/cli-command/src/intelliStory.js:28 (patternToRegex)
  • Severity: Medium
  • Reviewer: stack:code-reviewer (first pass); verified
  • Issue: With { globstar: true }, glob-to-regexp('*.css') compiles to ^([^/]*)\.css$, so bailOnChanges: ['*.css'] never matches src/Button.css. This is standard glob behavior — minimatch, picomatch, and GitHub Actions paths: filters agree — and the tests assert it deliberately (intelliStory.test.js:266, :286-289), so it is a considered choice rather than a coding error. It is listed because it lands on the same safety valve as findings 1 and 4, and there is currently no documentation for bailOnChanges/untraced anywhere in the repo.
  • Suggestion: Document that patterns are path-segment-scoped and **/*.css is required. If you prefer forgiving semantics, auto-broaden bare-name patterns: const p = pattern.includes('/') ? pattern : '**/' + pattern; — note this must be decided together with finding 1's rebasing.

7. noRequireBinding guard is narrower than the crash class it protects

  • File: packages/cli-command/test/noRequireBinding.test.js:40
  • Severity: Low
  • Reviewer: stack:code-reviewer (second pass)
  • Issue: FORBIDDEN = /\b(?:const|let|var)\s+require\s*=\s*createRequire\b/ only catches a single-line declaration. The same startup crash returns via import { createRequire as require } from 'module', const { default: require } = …, or a line-wrapped const require =\n createRequire(…) — none of which match.
  • Suggestion: Flag any binding of the bare identifier require (import specifier, declaration LHS, or parameter), not just the createRequire-RHS form.

8. mapEntry normalizes source only for type === 'src'

  • File: packages/cli-command/src/intelliStory.js:137
  • Severity: Low
  • Reviewer: stack:code-reviewer (first pass)
  • Issue: Fixtures only exercise 'src' and 'module' (the latter a no-op, since package specifiers are never absolute). If the enrichment tool ever emits a third type carrying an absolute path, that path ships unnormalized: it leaks a local or CI-runner path and cannot be correlated against project-relative affectedNodes.
  • Suggestion: Drop the type gate — resolveAndIndex already returns non-absolute values untouched, so running it for every string source is safe.

9. gitDiffNames omits --no-renames while getAffectedFileLocations uses it

  • File: packages/cli-command/src/intelliStory.js:75 vs :84
  • Severity: Low
  • Reviewer: stack:code-reviewer (first pass)
  • Issue: Rename detection is git's default since 2.9, so affectedNodes collapses a rename to the new path alone while getAffectedFileLocations (run with --no-renames) sees a delete + add pair. The two views of one diff disagree for renamed files.
  • Suggestion: Add --no-renames to gitDiffNames for parity.

10. path.relative does not case-fold when computing project-relative module ids

  • File: packages/cli-command/src/intelliStory.js:117-128 (resolveAndIndex)
  • Severity: Low (mechanism demonstrated, real-world trigger unconfirmed)
  • Reviewer: stack:code-reviewer (second pass)
  • Issue: path.relative('/Users/bob/proj', '/Users/Bob/Proj/src/App.js') returns '../../Bob/Proj/src/App.js' — exact segment comparison, no case folding, on every platform. On case-insensitive-but-preserving filesystems (APFS, NTFS), any casing divergence between the bundler's absolute module ids and git rev-parse --show-toplevel yields ..-prefixed index keys that can never match a real repo-relative path. The reviewer could not force an actual divergence, so this is flagged as plausible rather than confirmed.
  • Suggestion: At minimum, warn when the computed relative path escapes projectRoot (starts with ..) rather than silently indexing it; optionally normalize both sides through fs.realpathSync.

11. trace.html loads Drawflow from a public CDN

  • File: packages/cli-command/src/graphTraceTemplate.html:8, :12
  • Severity: Low
  • Reviewer: stack:code-reviewer — reported by both passes
  • Issue: The debug trace viewer pulls Drawflow's CSS and JS from cdn.jsdelivr.net at view time. Opt-in (trace: true) and off the production path, but it is precisely the artifact someone opens after a bail or mis-selection — often inside a locked-down or air-gapped CI runner where it renders blank.
  • Suggestion: Vendor the small Drawflow bundle and inline it into the template.

12. Minor: bail visibility, semgrep scope, stats-shape logging

  • Severity: Low
  • Reviewer: stack:code-reviewer (both passes)
  • Issue: (a) No bail logs at the throw site, so user visibility depends entirely on the @percy/storybook catch — outside this repo and still a follow-up per the PR body. (b) .semgrepignore:49-57 suppresses intelliStory.js file-wide rather than at the flagged joins, so future unsafe joins in this fs/path-heavy file go unscanned; the rationale is documented and the constraint (inline // nosemgrep unsupported by the CI version) is real. (c) readStats's stats.modules || [] (:161) makes a stats file with no modules key indistinguishable from a legitimately empty one.
  • Suggestion: (a) log.warn(message) inside a small bail(message, log) helper. (b) Track narrowing the suppression. (c) log.debug when the key is absent versus empty.

Informational — not counted as defects

  • devDependencies excluded from the lockfile gate (lockfileDiff.js:53-63, tested at lockfileDiff.test.js:85): Storybook addons are typically devDependencies and are imported directly by preview.js/story files. A pure addon version bump with no source diff will not surface through this path. It is explicit and tested, so likely intentional (relying on the server-side module graph instead) — worth a sanity check with the team given how conservative this feature aims to be.
  • stream-json leftover: confirmed absent from packages/cli-command/package.json and unreferenced — correctly removed in 9c525f10.
  • Backward compatibility: the new snapshotSchema fields (intelliStory, storybookPath) are optional, and createSnapshot sends 'intelli-story': intelliStory || null / 'storybook-path': storybookPath || null, so non-IntelliStory callers are unchanged — cli-upload/test/upload.test.js pins exactly that.
  • getStatus response shape: intelli_story_graph returns a top-level { status, data }, unlike the batched-by-id shape used for snapshot/comparison. pollGraphStatus and client.test.js:890-905 agree on it; an API-contract quirk, not a defect.

Verdict: FAIL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants