test(plugins/agento11y): drive the real Claude Code and Codex CLIs against a fake model - #771
Open
adamshovav wants to merge 2 commits into
Open
adamshovav wants to merge 2 commits into
adamshovav wants to merge 2 commits into
Conversation
…to land A headless `claude -p` run can fire Stop before its transcript writes land. Depending on timing, the file does not exist yet, or ends at the turn that called a tool: readTranscriptSettled either fails on the open or settles on that complete tool-use turn, and the SessionEnd that would catch up is cancelled when -p exits. The session exports nothing, or only the tool call. With CLAUDE_CODE_EAGER_FLUSH set, as in sessions the Claude desktop app hosts, the writes land in time, which hides the race. A Stop means the turn is over, so on Stop the settle read also waits, within the existing window, until the last complete assistant turn ended the turn rather than stopping for a tool, and treats a missing file as not yet written. Other events are unchanged, so a SessionEnd for a session that never wrote a transcript does not delay the exit.
…ainst a fake model TestAgentIntegration runs each CLI headless against a fake server for the Anthropic Messages and OpenAI Responses APIs, then checks the export with TestGoldenIntegration's capture server, secret guard, and invariants. The golden scenarios replay hook payloads written by hand; here the agent produces them, so a release that changes a payload, a transcript, or when a hook fires fails the run. The setup and checks the two tests share move out of runGoldenScenario, unchanged, into startHookExport, checkExports, and newLoopbackServer. CI installs the CLI versions pinned in testdata/agent-clis with npm ci and runs the test in the plugins/agento11y Go module job. Refs grafana#146
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #146. The first commit is #769, which the Claude run needs (#768); the harness is the second.
TestAgentIntegrationloads each plugin from this checkout, runs the CLI headless against a fake model on loopback, and checks the export with #190's capture server, secret guard, and invariants. The golden scenarios replay hook payloads written by hand. Here the agent writes them, so a release that changes a payload, a transcript, or when a hook fires fails the run.Run. One prompt carrying a Grafana Cloud token, one
echotool call, its result, and a final reply. Dummy keys and a throwaway HOME; it passes with outbound network blocked.claude -p --plugin-dir plugins/claude-codewithANTHROPIC_BASE_URL. Checks 1 export, 2 generations, the session ID, tool call, tool result, reply, and redacted token.codex plugin marketplace addandcodex plugin add, thencodex execwith amodel_providersentry. Checks 1 export, 1 generation, the tool call and result IDs, reply, and redacted token.The fake answers the Anthropic Messages and OpenAI Responses streaming APIs: a tool call first, then the reply once a request carries the call ID back.
Reuse.
startHookExport,checkExports, andnewLoopbackServermove out ofrunGoldenScenarioandnewGoldenServerunchanged. The golden tests pass as before.CI. The CLIs are pinned in
testdata/agent-clis: package.json and lockfile, installed withnpm ci, listed in the dependency-pinning check, withallowScriptsfor claude-code's postinstall to match the pnpmallowBuildspolicy. The step runs in theplugins/agento11yGo job and adds about 25s. zizmor reports nothing new. Codex runs with its sandbox off because it can't nest inside the runner's container; the only command the fake issues isecho.Locally it's opt-in:
The test clears
CLAUDE*,ANTHROPIC_*,CODEX_*, andOPENAI_*first: started from inside an agent session, the CLIs inherit variables that change the run, andCLAUDE_CODE_EAGER_FLUSHalone hides #768.Checked. Dropping tool results from the Claude mapper, the tool call ID from the Codex mapper, or prompt redaction fails the run. The full CI matrix passes on this branch. With #770, the Claude run drops from about 4s to under 1s.