feat(ci): run coder-eval through the published action, and publish an expiring evalboard link - #2942
Conversation
…expiring evalboard link Three things, all in the Linux job of the dispatch workflow. **Dogfood the action.** `UiPath/coder_eval@v0` had no real consumers: it was exercised only by its own repo's self-test, so a bad release would surface there rather than in someone's workflow. This job now installs and invokes coder-eval through it. What blocked adoption before is fixed upstream: the suite lives in `tests/`, needs an agent extra, and for delegate-sdk needs a plugin inside the environment the CLI actually runs from, and none of that was expressible. The single 150-line run step becomes prep + invoke. `working-directory:` is illegal on a `uses:` step and a job-level `defaults.run` does not reach inside a composite action, so everything conditional resolves into plain inputs first. The prep body is the head of the old step moved rather than rewritten: each `export FOO=bar` became a line in an env-block accumulator, and the trailing `coder-eval run` is gone. Every guard is preserved and was exercised locally against the extracted script: the three `:?` model hard-failures, delegate's `-j` clamp with its notice, its per-key `~/.uipath/.auth` validation, and the non-empty-only re-export of the three `*_VAR` knobs. All twenty non-`_VAR` env entries are forwarded, including the four model variables the shell also reads, because coder-eval's docker driver passes `CODEX_MODEL` and `ANTIGRAVITY_MODEL` into task containers by default. The bracketed `-D sandbox.docker.env_passthrough_extra=[...]` override now goes through the action's `args` input, which appends one argument per line verbatim. That is the same protection the old array-and-quote gave it, for the same reason: to bash the list is a character class. **An evalboard link per run.** The run uploads to a dedicated `runs-gha` container, which expires it after 14 days, and the link lands in the job summary. Entirely best-effort: every step warns and returns 0, because a missing dashboard link must never be mistaken for a failed eval. Inert until `AZURE_EVAL_UPLOAD_CLIENT_ID` exists as a repo variable, so this behaves exactly as before until the Azure identity is provisioned. Credentials are OIDC, not a storage key: `workflow_dispatch` runs the workflow definition from whatever branch the dispatcher picks, so a key in a repo secret would let any branch delete months of nightly history on that account. **Two fixes to things already broken.** The workflow had no redaction step, unlike smoke-skills.yml, so its artifact has been shipping unredacted transcripts; it is ported into both jobs and covers every secret this job forwards, not just the three smoke-skills strips. And the Linux artifact globs were off by one directory level, uploading no experiment- or variant-level report at all, because `--run-dir /tmp/runs` makes that directory itself the run dir. `run-name` is one line and independently the highest value per character available here: all ~2,500 historical runs are titled "Run Coder Eval". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rage credential The evalboard upload minted a GitHub OIDC token and exchanged it for an Azure credential, which needed a new Entra app registration, a federated credential scoped to a GitHub environment, and a container role assignment before it could work at all. The ADO pipelines already authenticate to this same storage account with `AZURE_STORAGE_KEY` / `AZURE_STORAGE_ACCOUNT`, and eval-runner passes that value straight to `BlobServiceClient`, so reusing it removes every one of those provisioning steps: two repo settings and nothing else. The Azure SDK sniffs the credential string, so the secret holds either the account key or a container-scoped SAS token with no change here. The SAS is preferable, since an account key is authority over the whole account including the nightly-history container, and this workflow runs its definition from whatever branch the dispatcher picks. Drops `id-token: write`, the `eval-upload` environment and the token-minting step. The gate moves to a job-env boolean because the `secrets` context is unavailable in any `if:`; deriving a boolean rather than exporting the secret keeps the credential in the upload step alone, matching how ADO scopes it. The container name is now hardcoded, since the unlisted evalboard source and the 14-day expiry rule both depend on ad-hoc runs never landing in `runs`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…on's args Follows the action's input simplification: it no longer promotes any of `coder-eval run`'s flags to named inputs, so the task globs and `--model` join the `-e`, `--type`, `-j`, `-v` and `-D` entries in the single `args` block the prep step already composed. The `model` step output goes away with them. Task globs are word-split when appended (TASK_GLOBS is a space-separated list from the partition job) but never pathname-expanded, which is an improvement on what the action used to do with them: the CLI expands them itself, so `**` works without `globstar` and a glob matching nothing exits 1 rather than reaching the CLI as a literal path. `step-summary: "false"` and `junit-path` are gone. The action writes no job summary at all now and reports `run-md-path` instead, so the append can happen after redaction, which is the order this workflow needs and previously had to work around. The append reads that output with the literal path as a fallback, because a composite action's outputs may not propagate from a step that exited non-zero and this step runs on failure. The JUnit report moves to /tmp/runs/junit.xml, derived from run-dir; nothing consumed it at the old path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y run The `v0` tag still serves the ten-input action, so this workflow's `with:` block would have `args`, `extras`, `extra-packages` and `working-directory` silently dropped — GitHub ignores an input the referenced tag does not declare. Pinning the PR head validates the whole path before anything merges and before Release moves `v0`. REVERT TO @v0 BEFORE MERGING. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comments only. Fifteen blocks cut to their contract, 118 lines out for 58 in, so the file sits at its existing 33% comment density instead of above it. Also drops a stale claim: the two-step-split banner said TASK_GLOBS "is handed straight to the action's `tasks` input, which word-splits it". There is no `tasks` input, and `args` entries are passed verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The link was appended after run.md, which is roughly 40 lines of tables, so it landed below the fold and was easy to miss. Prepend rather than reorder the steps: the evalboard step moves /tmp/runs, which both the artifact upload and the verdict step read, so it has to stay last. $GITHUB_STEP_SUMMARY is an ordinary file, so the block is written to a temp file with the existing summary appended, then truncate-and-rewritten in place to keep the inode the runner tracks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @bai-uipath's task in 3m 3s —— View job PR Review: feat(ci): run coder-eval through the published action
SummaryThis PR migrates the Linux coder-eval job from an inline Change-by-Change Review1. Temporary SHA pin — must revert before mergeSeverity: Critical
uses: UiPath/coder_eval@a747ce353111beadd51a6370b9e8e2dba88f4133 # bai/action-inputs-and-gha-sourceThe comment on line 622–623 says 2. Prep step uses
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | N/A | No skill files changed |
| E2E Tests | N/A | CI workflow change only |
| Skill Body | N/A | No skill files changed |
| References & Assets | N/A | No skill files changed |
| Repo Hygiene | OK | No secrets committed; credential scoping is well-designed |
Issues for Manual Review
- Upstream dependency: This PR is blocked on UiPath/coder_eval#147 being merged AND released to
v0. Dispatching before that release would silently run with no inputs (GitHub ignores unknownwith:keys on a composite action). AZURE_STORAGE_KEYprovisioning: The evalboard feature is inert until this repo secret is set. The PR is safe to merge before that — all evalboard steps gate onEVALBOARD_UPLOAD_ENABLED.eval-runner uploadCLI contract: The uploader's--adhoc,--title, and--descriptionflags come from thecoder-eval-uipath-eval-runnerpackage. If that package's CLI surface changes, this step breaks silently (continue-on-error: true). Worth confirming the contract is stable.
Conclusion
This is a well-engineered PR with thorough attention to security (credential scoping, redaction, least-privilege permissions), failure resilience (every evalboard step is best-effort), and backward compatibility (evalboard is inert without the secret, artifact paths are fixed not changed). The code is heavily commented with clear rationale for every decision.
One Critical item: revert the temporary SHA pin at line 624 to @v0 before merging. The author has already flagged this. Everything else is clean.
The previous attempt prepended to $GITHUB_STEP_SUMMARY inside the evalboard step. That cannot work: the variable points at a SEPARATE file per step, and GitHub builds the job summary by concatenating them in step order, so the step read its own empty file and its block still rendered last. Verified on run 33564007553, where the link stayed at the bottom. Move the publish above the step that appends run.md, which is the only way to control position, and copy the run directory instead of moving it so the artifact upload and the verdict still find /tmp/runs. Also emit a ::notice::, which renders above the job list on the run page. Keep "Run Coder Eval" as the run-name prefix so the run list stays scannable by workflow as well as by glob, agent and actor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Code reviewMulti-model review (Gemini 3 + an Opus sub-agent + my own reading). The disputed GitHub Actions semantics are verified against GitHub's docs and against the Critical1. # TEMPORARY — REVERT TO @v0 BEFORE MERGING. `v0` still serves the old
# ten-input action, which would silently ignore every input below.
uses: UiPath/coder_eval@a747ce353111beadd51a6370b9e8e2dba88f4133I checked both action versions. So the note as written instructs a change that breaks the workflow: GitHub only warns on unexpected High2.
So The clean fix is to stop moving the directory: compute 3.
Fix: an explicit 4.
5. Redaction touches only Medium6. Confirmed in the coder_eval repo ( Worth noting 7.
8. The comment at 454 says "Values must be single-line," but nothing checks. A secret rotated to a multi-line value splits into two entries and the action parses the second as another case "${!name}" in *$'\n'*)
echo "::error::$name contains a newline; the env block is single-line only"; exit 1 ;;
esacSeparately, the delimiter-collision risk motivating the randomized delimiter (602) is essentially unreachable — that comment does more work than the risk warrants. 9. Plus a third near-copy at 10. "JUnit lands at /tmp/runs/junit.xml and travels with the run into the artifact and the blob." None of the upload patterns at 743-756 match 11. The limit is per step, and per the docs upload failures for job summaries don't affect step or job status. An oversized 12. The premise is right (an omitted permission becomes Low
What's good here
OverallThree real fixes here — the artifact-path off-by-one, the missing redaction, and a better-structured invocation — but the headline feature doesn't work: 🤖 Generated with Claude Code |
uipreliga
left a comment
There was a problem hiding this comment.
Fix what you agree with and 🚢
`--run-dir /tmp/runs` makes /tmp/runs the run dir itself, so its subdirectories are VARIANTS. The verdict step selected one with `ls -td /tmp/runs/*/ | head -n 1`, which is correct only while nightly.yaml declares a single variant: a second one would silently grade a single arm and could report a red run green. This is the same off-by-one the artifact globs in this branch already fix, one step lower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The temporary SHA pin pointed at the PR branch of coder_eval#147, which is now merged and released as 0.11.6 with v0 promoted to it, so the eight-input surface this workflow passes is what v0 serves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Consumer half of giving
run-coder-evaldispatches a shareable evalboard link while moving the Linux job onto the published composite action. Upstream half: UiPath/coder_eval#147.What this does
Dogfoods the action. It had no real consumers: only its own repo's self-test, so a bad release surfaces there instead of in someone's workflow. What blocked adoption was concrete, and is fixed upstream: the suite lives in
tests/, needs an agent extra, and for delegate-sdk needs a plugin inside the environment the CLI actually runs from.Being the first real consumer is also what reshaped the action. It promoted five of
coder-eval run's 21 flags to named inputs with no principle behind the choice, and since GitHub silently ignores an unknown input, every forwarding input is a way for a run to measure something else and still exit 0. The upstream PR drops all of them, so the whole command line arrives through oneargsblock — the task globs,--model,--type,-j,-vand the-Doverrides, all composed by theprepstep that already existed here.That makes this side simpler, not more complex. The
modelstep output goes away.step-summary: "false"goes away because the action no longer writes a job summary at all: it reportsrun-md-pathand the append happens after redaction, which is the order this workflow needs and previously had to work around with a comment explaining why the input could not simply be turned on. Globs also stop being shell-expanded, so**works withoutglobstarand a glob matching nothing exits 1 instead of reaching the CLI as a literal path.Emits an evalboard link per run, into a dedicated
runs-ghacontainer that deletes it after 14 days. No Slack, no new dashboard tab, no pollution of nightly history.Fixes two things that were already broken and would have looked like my doing once I rewrote the neighbouring steps. See below.
The refactor, and why it is safe
The single 150-line run step becomes prep + invoke, because
working-directory:is illegal on auses:step and a job-leveldefaults.rundoes not reach inside a composite action. The prep body is the head of the old step moved, not rewritten: eachexport FOO=barbecame a line in an env-block accumulator, and the trailingcoder-eval runis gone. Read it as a diff against the old step.Since a paired dispatch is not yet possible, I extracted the prep script straight out of the YAML and drove it locally through ten scenarios. All ten behaved:
--type, extras, argsCODEX_MODEL/ANTIGRAVITY_MODEL/CLAUDE_CODE_MODELunsetDELEGATE_MODELunsetkimi-k2-7-codewithout failingparallelism: 20on delegate::notice::fires,-j 6reaches the action~/.uipath/.authmissing one key::error::+ exit 1, per key*_VARknobs-DoverrideAll twenty non-
_VARenv entries are forwarded, including the four model variables only the shell reads. That last part is not defensive padding: coder-eval's docker driver passesCODEX_MODELandANTIGRAVITY_MODELinto task containers by default, so dropping them because the shell also reads them would have been a real change.The bracketed
-Doverride goes through the action's newargsinput, which appends one argument per line verbatim, preserving exactly what the old array-and-quote protected it from.The evalboard path
Best-effort by construction. Every step in that block warns and returns 0, and the whole block is skipped until
AZURE_STORAGE_KEYis set as a repo secret, so this merges and behaves exactly as before while the credential is still being provisioned. Four failure paths tested locally: norun.json, uploader absent, upload fails, happy path. All exit 0; only the last one prints a link.Authentication reuses the credential the ADO pipelines already use for this storage account (
AZURE_STORAGE_KEY/AZURE_STORAGE_ACCOUNT, out ofcoder-eval-athena-{secrets,config}) rather than standing up a second, GitHub-only Entra app with a federated credential. eval-runner passes the value straight toBlobServiceClient, and the Azure SDK sniffs the string, so the secret can hold either the account key or a container-scoped SAS token with no change to this workflow.A scoped SAS is the better thing to put there.
workflow_dispatchruns the workflow definition from whatever branch the dispatcher picks, and an account key is authority over the entire account, including therunscontainer that holds months of nightly history. A SAS limited toruns-ghawith create/write bounds the blast radius to the container this feature owns; the cost is an expiry to rotate.Two things narrow the exposure regardless: the credential is set only on the upload step, never job-wide, which matters because the steps before it run an agent with shell access (the ADO pipelines scope it the same way and say so); and the container name is hardcoded with no variable override, so no repo setting can redirect an upload into
runs.permissions:is job-level onrun-linuxonly. Apermissions:block makes every permission it omitsnonefor every job it covers, and this file declares none today, so a workflow-level one would have silently stripped the Windows job's token.Already broken, fixed here
smoke-skills.ymlhas had one since the reports started carrying agent transcripts; this workflow never did, so its artifact has been shipping them unredacted. Ported into both jobs, covering every secret this job forwards rather than the three smoke-skills strips, plus the ROPC token read from disk. Runs after the.venv/node_modulescleanup so it does not walk tens of thousands of files for nothing.--run-dir /tmp/runsmakes that directory itself the run dir, so/tmp/runs/*/experiment.htmlmatched nothing and no experiment- or variant-level report has ever been uploaded. The Windows copies are correct as written, because that job passes no--run-dir.Deliberately not done
minimum-task-scoreis not adopted. The gate here is "every row is SUCCESS", which is a different predicate from aweighted_scorefloor, and swapping gate semantics inside a behaviour-preserving refactor would make a regression indistinguishable from the intended change.run-md-pathand writes nothing itself, which is what lets the append land after redaction. It reads that output with the literal path as a fallback, because a composite action's outputs may not propagate from a step that exited non-zero and this step runs on failure. Same 1 MB cap: GitHub rejects a summary over 1 MiB outright, losing the verdict with it.python | teepipeline would have become the step's exit status and turned every red Windows run green.Verification
actionlint(with the two self-hosted labels configured) is clean, and shellcheck reports fewer findings thanorigin/mainfor this file: the two unquoted-expansion warnings in the old run step are gone and nothing new was added. Everyrun:body parses underbash -n, and both embedded Python blocks compile.Azure side is already in place: container
runs-ghacreated private, and lifecycle ruleexpire-runs-gha-14d(prefixMatch: runs-gha/, 14 days) appended to the account policy without disturbingexpire-runs-live-7d.End-to-end, on a temporary SHA pin
uses:cannot take an expression, so both dispatches below pinnedUiPath/coder_eval@a747ce35(the upstream branch head) in place of@v0. That pin is reverted before this merges and is marked as such in the file.Both ran
tasks/uipath-automationhub/*.yamlat-j 2, with the Windows job correctly skipped (neither task carries thewindowstag). Confirmed from the logs: thetests/.coder-eval-versionpin resolved (coder-eval==0.11.5), thelitellmextra composed into the install requirement rather than being added afterwards (litellm==1.98.0),/tmp/runs/junit.xmland/tmp/runs/run.mdwere written and reported as outputs, and the evalboard link resolved against the SAS now held inAZURE_STORAGE_KEY.The delegate-sdk run is the one that exercises
extra-packages:coder-eval-uipath==0.1.0installed into coder-eval's own tool environment and itscoder_eval.pluginsentry point registereddelegate_sdk_agent. That discovery is only possible when the plugin shares the CLI's virtualenv, which is the entire reason the input exists. Its twelve-name-D sandbox.docker.env_passthrough_extra=[...]override also arrived as one intact argument throughargs.Still outstanding before a dispatch produces a link: the
AZURE_STORAGE_KEYrepo secret on UiPath/skills. OptionallyAZURE_STORAGE_ACCOUNTas a repo variable, which defaults tocoderevaltests. No Azure-side provisioning is left: theruns-ghacontainer and itsexpire-runs-gha-14dlifecycle rule already exist.🤖 Generated with Claude Code