chore(ci): enable the agentic Open Source fix [AG-387]

[ShawkyZ]

What this does

Enables the agentic Open Source fix in the ProdSec orb for the code-analysis job. One parameter:

open-source-agentic-fix-enabled: true

When the Enhanced Gate blocks an Open Source scan, snyk fix --agentic runs against the blocking vulnerabilities, commits the result to <branch>+remy_fix, and opens a pull request against the branch that failed — so the next action is reviewing a diff rather than reading a CVE.

Released in prodsec-orb v1.2.29, so no version bump is needed: @1 already resolves to it. main already attaches the prodsec-orb-runtime context that supplies the credentials, which is why this is a single line.

Nothing else changes. open-source-scan stays at high, so what blocks this repository is exactly what blocks it today.

Why it's low risk

• The fix only runs after the gate has already failed the build. It cannot change that verdict — every path through the step exits 0, so it can neither mask a failure nor turn a passing build red.
• It never pushes to this branch. Work goes to a separate +remy_fix branch and arrives as a pull request for review.
• It refuses to act on anything unexpected: no fix branch if the tree is already dirty, no commit of build output, lockfile-only staging, and hard caps on the size of a change it will push.
• Disabling it is deleting the line.

One caveat worth knowing rather than discovering: enabling the feature adds a github-cli/install step that runs on every build of this repository, not only blocked ones, because a CircleCI step cannot be made conditional on a runtime value. If that install fails it fails the job — the one new way a green build could go red.

Also note snyk fix runs this repository's dependency lifecycle scripts in the same job that holds the context secrets. See the orb's Agentic Fix docs for the full security discussion.

🤖 Generated with Claude Code

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[cursor]

Stale comment

Automated verification (/verification)

Scope: .circleci/config.yml only — CI/CD config change (CircleCI orb pin swap, security-scan gate change, agentic-fix trial). Classified as CODE (non-Markdown), so the full reviewer panel ran: semantic-analysis, adversarial-review-runner, security-scan, code-reviewer.

Verdict: Changes requested. Two independent lenses (adversarial-review, security-scan) each independently raised the same Critical-severity gap, so this is not auto-approved.

Critical

The PR description already discloses the top-line risks (mutable/expiring dev-orb pin, github-cli/install now running on every build, LLM/Snyk-token visibility to snyk fix's lifecycle scripts) — those are not re-raised here. What both agents flagged as an additional, unaddressed gap:

• The PR's safety claim that REMY_GITHUB_TOKEN (repo-write scoped) "is read once and unset before dependency-lifecycle scripts run" is implemented inside the mutable dev:a84230eb... orb build itself — the very artifact this PR already flags as mutable/unaudited (.circleci/config.yml:47). Anyone with publish rights to that dev-orb label can ship new content under the same tag with zero corresponding diff in this repo, which would run automatically on the next code-analysis build (every non-release branch) and could exfiltrate LITELLM_API_KEY/LITELLM_BASE_URL/the GH token before any unset step runs, or skip the unset step entirely. This turns "someone with orb-publish rights can break my CI" into "...can silently exfiltrate a repo-write GitHub token and LLM credentials from every non-release build."
• Relatedly, enabling open-source-agentic-fix-enabled: true (:1532) alongside that same repo-write token has no in-repo scoping or approval gate on what the agent can do with it (e.g. nothing here constrains it to opening a PR vs. pushing directly) — that control is entirely delegated to orb-internal behavior this repo can't audit.

Should Fix

• Dev-orb expiry (90 days) has no automated tripwire — if the revert is missed, all CI for the whole repo breaks at config-compile time, not just this feature (:47).
• code-analysis is wired as a shared, all-branches job (filters: branches: ignore: '/release.*/'); this same file already has a narrower-scope pattern (sign jobs use filters: branches: only: [...]) that wasn't used for this trial (:594).
• No single toggle/pipeline-parameter exists to disable the trial — reverting means editing 4 separate sites instead of one (:1532).
• open-source-scan: medium has no revert/expiry tracking of its own, unlike the orb pin — if the orb pin is reverted without also reverting this line, the loosened gate becomes permanent by default (:1530).
• Scan-severity change and the agentic-fix flag are enabled in the same commit, so a post-merge CI regression can't be attributed to a single cause (:1530).

Suggestion

• The comment "Without this the feature skips, naming what is missing" (:600) states unverified orb-internal behavior as fact; the PR's own "Not verified" section says this path has never completed end-to-end successfully anywhere yet.

What's good

• Confirmed: YAML parses cleanly, no duplicate keys, and prodsec-orb-runtime appears exactly once, scoped only to code-analysis — no leakage to other jobs/workflows.
• Change is compact (~4 semantic edits) and easily revertible.
• Inline comments explain rationale well, and the PR description is unusually transparent about known tradeoffs — that materially narrowed this review's scope.

Policy note

Per this repo's automated-review policy, auto-approval is reserved for dependency-bump-only changes (e.g. go.mod/go.sum, package-lock.json). This PR modifies CI/CD configuration and introduces a new secrets surface (prodsec-orb-runtime: LLM credentials + a repo-write GitHub token) tied to a mutable, unreleased third-party orb build, so it is intentionally not auto-approved here. Given the token/supply-chain surface, recommend an explicit human security sign-off before merge, ideally after addressing the branch-scoping and revert-tracking gaps above.

  

_{Sent by Cursor Automation: Automatic PR verification}

[cursor]

[Critical] Enabling the agentic fix with a repo-write-scoped token (REMY_GITHUB_TOKEN via prodsec-orb-runtime) has no in-repo scoping or approval gate — nothing here constrains the agent to opening a PR vs. pushing/merging directly; that control is entirely delegated to unaudited orb-internal behavior.

Fix: confirm and, where possible, encode in this config (e.g. a required manual-approval step) that the agentic fix can only open a PR for human review, never push/merge directly with this token.

[cursor]

Stale comment

/verification results — not auto-approved

Approval policy check: this automation auto-approves only when a PR is a pure dependency-version bump (e.g. go.mod/go.sum, package.json/package-lock.json). This PR's only changed file is .circleci/config.yml — a CI pipeline configuration change, not a dependency bump — so it is not eligible for auto-approval regardless of the findings below.

Ran the full four-lens verification (semantic analysis, adversarial review, security scan, code review) against the diff. The author's own "Please read before approving" section already discloses several tradeoffs (medium + ignore-SLA compounding, the unconditional github-cli/install step, the token-unset timing). The findings below are things that section does not cover, deduplicated across the four reviewers.

Critical

1. Orb reference doesn't match what the PR claims was validated (.circleci/config.yml:47). The PR description's change table says @1 → @dev:ee7f5290, but the committed diff pins @dev:a84230ebf9a6fc3aba9b84d41f896a6e858a5ec6 — a different dev-orb label. Dev labels are mutable snapshots, so the cited safety evidence ("circleci config validate passes", "82 green checks on the orb side") may describe a different build than the one that will actually execute with repo-write credentials against this repo. Two independent reviewers caught this by cross-checking the PR body against the diff.
2. The disclosed token mitigation doesn't cover the step that runs first (.circleci/config.yml:598-601, job at 1491). prodsec-orb-runtime (supplying LITELLM_API_KEY, LITELLM_BASE_URL, LITELLM_MODEL, REMY_GITHUB_TOKEN) is attached at job scope, so it's ambient to the entire code-analysis job from job start — including setup-go-private-modules and make lint (line ~1518-1520), both of which run before prodsec/security_scans (line 1521) ever executes. Whatever "read once, unset, passed only as a per-command env prefix" logic the orb applies to the token happens inside that later step — it protects nothing that ran earlier. make lint's process tree (eslint/prettier plugins, etc. — arbitrary third-party JS) has full ambient access to a repo-write GitHub token for its entire duration.
3. Feature enabled unconditionally with zero known successful runs (.circleci/config.yml:1537). Per the PR body itself, the agentic fix "has never completed end-to-end against a real LLM, a real gh install, or a real authenticated push," and a parallel trial on snyk-intellij-plugin#880 "has not yet produced a fix branch, and that failure is still undiagnosed." This PR turns the same, still-unproven mechanism on for every branch push (see #5 below) in the flagship public CLI — a much larger blast radius than the sibling trial, without adding diagnostics the other trial lacks.

Should Fix

4. Mutable dev-orb pin in the shared release pipeline (.circleci/config.yml:47). Anyone with push access to prodsec-orb's dev branch can change what this repo's CI executes with no corresponding commit/diff/review here — and finding #1 shows this kind of drift already appears to have happened once during this PR's life. No expiry date or tracked ticket is recorded next to the pin to force the revert before the ~90-day expiry.
5. Trial is scoped to (almost) every branch, including main (.circleci/config.yml:604-607). The filter excludes only /release.*/, so the untested agentic-fix path and its credentials apply to commits landing on the default branch, not a controlled trial population.
6. open-source-scan: medium (line 1535) applies unconditionally, not just when the agentic-fix path is exercised — it widens the CI gate for every contributor's branch for the duration of the trial, and increases how often the secret-exposure window in finding #2 is reached.
7. No opt-in gate used, despite an existing pattern in this same file — .circleci/config.yml already has a boolean pipeline-parameter + when: pattern (e.g. the static parameter, ~lines 178-184) that could scope this entire trial to manually-triggered runs instead of every push to every branch. This is the highest-leverage fix available: it preserves the ability to test the feature while removing "every push, every branch" as the default population.

Suggestions

8. no-output-timeout: "30m" (line 1527) gives a compromised orb step or subprocess 3x longer to run silently (network calls, pushes with the repo-write token) before CircleCI's own liveness safeguard would intervene. Lower risk once 2/5/7 above are addressed.
9. Running this trial in parallel with the already-failed, undiagnosed snyk-intellij-plugin#880 trial doubles unexplained exposure instead of sequencing risk down — diagnosing the cheaper failure first would be more informative.
10. Whether fork-PR builds run this job with contexts/secrets attached is a CircleCI project setting not visible in this YAML — worth confirming operationally, since if so, finding #2's exposure would also be reachable from external, unreviewed contributor code.

What went well

• prodsec-orb-runtime was deliberately confirmed to land only on code-analysis, not sprayed across the workflow.
• Unusually thorough inline documentation of why each change was made, and a genuinely self-aware "please read before approving" section that made this review more productive.

Unverified claims

Several security-relevant claims (exactly how the orb handles REMY_GITHUB_TOKEN internally, whether snyk fix --agentic truly invokes npm/go lifecycle scripts) come from the PR description only — snyk/prodsec-orb is private and could not be independently inspected. These are treated as open gaps, not verified mitigations.

Recommendation: address the orb-ref/PR-description mismatch (#1), reduce the credential/lint co-location risk (#2), and adopt an opt-in trial mechanism scoped away from main (#5, #7) before merging.

  

_{Sent by Cursor Automation: Automatic PR verification}

[cursor]

Critical. This pins @dev:a84230ebf9a6fc3aba9b84d41f896a6e858a5ec6, but the PR description's own change table says the move is to @dev:ee7f5290 — a different dev-orb label. Dev labels are mutable snapshots, so the PR's cited safety evidence (circleci config validate passes; 82 green checks on the prodsec-orb side) may not describe the exact build that will execute here with repo-write/LLM credentials attached. Please re-validate against the exact committed label (or update the pin to match whatever was actually validated) and reconcile the PR description.

[cursor]

Verification result: not auto-approved

Per policy, this bot only auto-approves PRs that are pure dependency-version bumps (e.g. a go.mod/package.json/lockfile bump with no other change). This PR's only file is .circleci/config.yml, and while it does re-pin the prodsec orb version, it bundles that with three additional behavioral changes: lowering the open-source-scan severity gate from high to medium, adding a new secrets-bearing CircleCI context (prodsec-orb-runtime) to a job, and enabling a brand-new open-source-agentic-fix-enabled feature flag. That is not a dependency bump — it's a CI security-gating and trust-boundary change — so it is not eligible for automatic approval and requires a human reviewer's sign-off.

I ran the full four-lens verification pass (semantic analysis, adversarial review, security scan, code review) against the diff. All four lenses independently reached the same "not a pure dependency bump" conclusion, and surfaced the findings below. Two Critical items are new since the last automated pass; the rest refine or reconfirm points from the author's own "please read before approving" section and prior automated reviews on this PR.

Progress since the last run: the newest commit already removed prodsec-orb-runtime from the secrets-scan job's context, narrowing that job's secret exposure. Good change — see remaining scoping concern on code-analysis below, which still carries the same class of risk.

Critical

1. Job-scoped secrets reach untrusted build steps (.circleci/config.yml:601) — prodsec-orb-runtime (carrying LITELLM_API_KEY, LITELLM_BASE_URL, LITELLM_MODEL, REMY_GITHUB_TOKEN) is attached to the whole code-analysis job. CircleCI contexts export secrets for every step in the job, and code-analysis also runs make lint against this repo's own (PR-branch-controlled) Makefile/tooling before the security-scan step executes — with the secrets already in the environment. A change to lint tooling in a PR could read and exfiltrate these without ever touching the agentic-fix code path.
2. Repo-wide CI trust boundary now depends on a mutable, unreleased artifact (.circleci/config.yml:47) — snyk/prodsec-orb@dev:<sha> is a dev-channel reference, not an immutable release; its content can change without any corresponding commit/PR/review in snyk/cli. If the label is ever deleted, mutated unexpectedly, or simply expires (see below), CircleCI config compilation can fail for every job in the repo, not just the ones exercising this trial.

Should Fix

3. Severity gate widened for every branch, not just the trial (.circleci/config.yml:1530) — open-source-scan: medium (kept alongside the pre-existing open-source-block-ignore-sla: true) applies to all code-analysis runs on all non-release branches, so unrelated PRs can start failing purely because the bar moved, independent of the agentic-fix trial itself. The author's own PR description flags this, but nothing scopes the threshold change away from the shared gate.
4. Orb's internal github-cli/install step now runs unconditionally on every build (affects the prodsec/security_scans call at .circleci/config.yml:1524-1533) — per the author's own description, this step can't be conditioned on the trial's runtime state, so a transient install failure now fails code-analysis for reasons unrelated to the security gate, on every branch.
5. No automated tripwire for the 90-day dev-orb expiry (.circleci/config.yml:46) — the revert plan lives only in a code comment tied to an external PR's release, with no calendar checkpoint or linked tracking issue to force the revert before the label silently stops resolving or changes content.

Suggestions

6. The three risky changes (orb pin, severity threshold, new context/flag) are hard-coded directly into the shared code-analysis job body rather than gated behind a pipeline.parameters toggle, making the trial harder to disable quickly if the agentic fix misbehaves mid-rollout (the author notes it has never run end-to-end successfully).
7. The revert-plan comment doesn't reference the AG-387 tracking ticket, so git blame/grep alone won't point a future reader at ownership or status.

Verdict: CONTESTED / not approved. The two Critical findings — job-wide secret exposure to untrusted lint code, and a mutable trust anchor on the shared CI gate — are, in my assessment, real and not fully covered by the author's existing mitigations (which address the GitHub token's use inside the orb, not its exposure to make lint running earlier in the same job). Recommend isolating the prodsec/security_scans step (and its secrets context) into its own job before merging, and scoping the severity-threshold change away from the default gate for unrelated PRs.

  

_{Sent by Cursor Automation: Automatic PR verification}

stale

[snyk-pr-review-bot]

PR Reviewer Guide 🔍

🧪 No relevant tests

🔒 Security concerns Secret Exfiltration Risk: The PR description acknowledges that snyk fix executes the repository's dependency lifecycle scripts (e.g., postinstall) within the same job that holds the prodsec-orb-runtime context secrets. If a malicious PR introduces a vulnerability (to trigger the fix) and a malicious script, the agentic fix process could potentially execute that script and exfiltrate the high-privilege credentials (like GitHub tokens) provided by the context.

⚡ Recommended focus areas for review Increased Pipeline Fragility 🟡 [minor] Enabling open-source-agentic-fix-enabled introduces a mandatory github-cli/install step that runs on every execution of the code-analysis job. As noted in the PR description, if this external installation fails, the entire job fails. This introduces a new external point of failure (GitHub CLI distribution) that can block the CI pipeline even when the underlying code is secure and passing tests. open-source-agentic-fix-enabled: true

📚 Repository Context Analyzed This review considered 1 relevant code sections from 1 files (average relevance: 0.70) packages/snyk-protect/README.md (lines 1-79)

🤖 Repository instructions applied (from AGENTS.md)