test: make drift output test independent of colour detection - #7057
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This comment has been minimized.
This comment has been minimized.
b72187a to
32c679b
Compare
This comment has been minimized.
This comment has been minimized.
| const originalLevel = chalk.level; | ||
| const originalEnabled = chalk.enabled; | ||
|
|
||
| beforeAll(() => { |
There was a problem hiding this comment.
Question: has this been an issue you experienced somehwere? Otherwise such a global state change will become a problem when we start parallelizing things. Maybe NO_COLOR=1 for the CLI subprocess makes more sense!
32c679b to
3f44d5e
Compare
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Stale comment
Verification Report
Ran the 4-agent verification pass (semantic analysis, adversarial review, security scan, independent code review) against this PR's diff (
test/jest/unit/lib/iac/drift/drift.spec.ts, +17/-0).Verdict: PASS — no findings from any agent.
- Semantic analysis: No findings. Verified against the actual
chalk@2.4.2source that pinningchalk.level = 1produces exactly the ANSI byte sequences infixtures/all.console, thatbeforeAll/afterAllcorrectly capture and restore the ambient state, and that Jest's per-file module isolation rules out cross-file leakage.- Adversarial review (Skeptic/Architect/Minimalist): No findings. Confirmed there's no terminal-detection logic in
getHumanReadableAnalysisbeing masked by this fix, and that forcing a deterministic chalk level is the correct fix for the described flakiness rather than a workaround.- Security scan: No findings — test-only change, no security-relevant surface (no manifest changes, no untrusted input, no new sinks).
- Independent code review: No findings. Minimal, correctly scoped fix; restoration runs unconditionally in
afterAllregardless of test outcome.Auto-approval policy
This PR's change is a test-only fix (
drift.spec.ts), not a dependency bump (e.g.go.mod/package.json/lockfiles). Per this automation's policy, only dependency-only changes are auto-approved — so no automatic approval was applied here, despite the clean verification pass. This is a neutral, comment-only review; it does not block or change the mergeability of the PR.Sent by Cursor Automation: Automatic PR verification
The "Test describe output" case compares getHumanReadableAnalysis against the all.console fixture, which contains hard-coded ANSI escapes. Whether those escapes are produced is decided by chalk from the ambient environment, so the test only passes when chalk believes colour is supported. chalk 2.4.2 ships supports-color 5.5.0, which returns level 0 for a non-TTY stream before it considers CI vendor variables. CircleCI runs steps under a pty so the test passes there, but piping jest output anywhere else -- a local run, an editor test runner, a container -- produces plain text and a large confusing diff. Pin chalk's colour level for this block and restore it afterwards, so the assertion tests the formatting rather than the terminal it happens to run in. Verified passing with no FORCE_COLOR, FORCE_COLOR=1 and FORCE_COLOR=0. Co-authored-by: Cursor <cursoragent@cursor.com>
3f44d5e to
470ba5a
Compare
PR Reviewer Guide 🔍
|
There was a problem hiding this comment.
Verification Report
Re-ran the 4-agent verification pass (semantic analysis, adversarial review, security scan, independent code review) against this PR's current head (470ba5a), diff unchanged in substance from the previous round: test/jest/unit/lib/iac/drift/drift.spec.ts (+17/-0).
Verdict: PASS — no blocking findings from any agent.
- Semantic analysis: No findings. Verified capture/restore ordering is correct (values captured in the
describebody beforebeforeAllruns), confirmedafterAllrestores state even if a hook throws, and confirmed via chalk 2.4.2 source thatchalk.leveldoesn't affect the two styles used here (bold/bluehave fixed codes) — so the fixture can't mismatch on level choice. Also empirically proved no cross-file leakage: ran this spec alongside another file that mutateschalk.enabledwithout restoring it, with--runInBand(worst case for leakage) — both suites passed independently, confirming Jest's per-file module isolation. - Adversarial review (Skeptic/Architect/Minimalist): PASS. The Skeptic empirically injected two regressions into the production code under test and confirmed both were caught (the fix doesn't weaken detection). The Architect directly investigated the global-mutation concern raised in the review thread below by @PeterSchafer — built a two-file leak probe and confirmed no contamination is possible across spec files; scoping to a single
describeblock withafterAllrestoration is correct. Only low-severity, non-blocking suggestions remain (e.g. the codebase's existingtoContainText/ANSI-stripping matcher solves a related-but-different problem and isn't a drop-in replacement here, since this test needs to verify exact ANSI placement). - Security scan: No findings — test-only change, zero manifest diffs, no secrets, no new attack surface.
- Independent code review: No findings. Confirmed both required chalk gates (
levelandenabled) are pinned (chalk'sapplyStylerequires both), and the fixture's raw bytes are basic ANSI (bold/blue), matchingchalk.level = 1exactly. - Local reproduction: independently ran
npx jest test/jest/unit/lib/iac/drift/drift.spec.tswith default (piped) output,FORCE_COLOR=0, andFORCE_COLOR=1— all 19 tests pass in every mode, confirming the fix resolves the flakiness as described.
Auto-approval policy
This PR's change is a test-only fix (drift.spec.ts), not a dependency bump (e.g. go.mod/package.json/lockfiles — none touched, confirmed via diff and the PR's own security/snyk check reporting no manifest changes). Per this automation's policy, only dependency-only changes are auto-approved, so no automatic approval was applied here despite the clean verification pass. This is a neutral, comment-only review; it does not block or change the mergeability of the PR.
Sent by Cursor Automation: Automatic PR verification


Pull Request Submission Checklist
What does this PR do?
Makes
test/jest/unit/lib/iac/drift/drift.spec.ts→Test describe output › test output for known analysispass regardless of the terminal it runs in.That test compares
getHumanReadableAnalysisoutput against thefixtures/all.consolegolden file, which contains hard-coded ANSI escapes (\e[1m…\e[22m,\e[34m…\e[39m). Whether those escapes are emitted at all is decided by chalk from the ambient environment, so today the test passes only when chalk happens to think colour is supported.Concretely: chalk 2.4.2 depends on supports-color 5.5.0, which returns level
0for a non-TTY stream before it gets as far as checking CI vendor variables. CircleCI runs steps under a pty, so the test passes in CI. Anywhere jest output is piped — a plain local run, an editor-integrated runner, a container, a sandboxed agent — chalk emits plain text and the test fails with a large and fairly baffling whitespace-looking diff:This wasn't hypothetical: it's reproducible on a normal dev machine with full network access and valid credentials, and it was originally reported as an environment problem because the failure looks nothing like a colour issue.
The fix pins chalk's colour level for that
describeblock and restores the previous value afterwards, so the assertion covers the formatting logic rather than the terminal it happens to run in.Where should the reviewer start?
The whole diff is ~16 lines in
drift.spec.ts. Worth confirming you agree with the direction: this keeps the golden file asserting on colours (rather than stripping ANSI from both sides, which would silently stop testing them).How should this be manually tested?
On
mainthe first and third fail. Fullnpm run test:unitwas also run: this suite passes and nothing else changed behaviour.What's the product update that needs to be communicated to CLI users?
None — test-only change, no shipped code touched.
Risk assessment: Low
Test-only. The chalk level is scoped to one
describeblock and restored inafterAll, and jest gives each test file its own module registry, so no other suite is affected.Made with Cursor