refactor(action)!: eight inputs, none of them a CLI flag, plus an unlisted evalboard source for ad-hoc GH runs

[bai-uipath]

Upstream half of giving UiPath/skills' run-coder-eval dispatch a real evalboard link while moving it onto the published composite action. Along the way the action's input surface got rewritten rather than extended. The skills-side PR consumes it and is blocked on this one being merged and released, since uses: UiPath/coder_eval@v0 reads its inputs from the action at that tag.

Why the input surface changed instead of growing

The action had no real consumers, only this repo's own action-dogfood job and the synthetic e2e in verify-published-action.yml, so a bad release surfaced in a self-test rather than in someone's workflow. Making run-coder-eval a real dependent meant adding inputs for a suite in a subdirectory, an agent extra, and a plugin inside the environment the CLI runs from. Five new inputs took the count to fifteen, and at that point the shape of the thing was the problem, not the gaps.

coder-eval run has 21 flags. The action promoted five of them to named inputs with no principle behind the choice: --tags got one, its sibling --exclude-tags did not, and extra-args's own description admitted it covered "--tags exclusions". A forwarding input buys nothing and costs a lot, because GitHub silently ignores an input the referenced tag does not define. One that is mistyped, or newer than the consumer's pin, yields a run that measured something else and still exits 0. A wrong CLI flag is a hard error instead.

The surface: 15 inputs to 8, none of them a CLI flag

An input now exists only where the action does something with the value besides pass it along.

Kept	Because
version, extras, extra-packages, install-flags	compose the install spec, which the action owns
working-directory	applied to the action's own steps; illegal on a uses: step, and job-level defaults.run does not reach into a composite
env	exported into the action's own shell, so a forwarded secret never touches $GITHUB_ENV
args	the single passthrough
run-dir	read back for the outputs

Removed	Where it went
tasks, tags, model, extra-args	all fold into args
prerelease	generalises to install-flags, which also covers --extra-index-url for a private index
junit-path	derived as <run-dir>/junit.xml, which is where both existing consumers already put it
step-summary	replaced by a run-md-path output. A consumer that has to redact the report first cannot undo a write that already happened, so the write is the consumer's call. cat "$RUN_MD" >> "$GITHUB_STEP_SUMMARY" is the whole of the default this replaces
minimum-task-score	deleted, not relocated — see below

Four different encodings (whitespace-split, line-delimited, comma-separated, NAME=VALUE) collapse to one clean_lines parser, copied into both step scripts because they are separate bash processes, with a test asserting the copies stay byte-identical.

Folding task globs into args removes a hazard

They now reach the CLI unexpanded and expand_task_files handles them. Verified directly:

recursive glob : 48 files      # **/*.yaml — no globstar needed
flat glob      : 21 files
non-matching   : exits 1       # "No task files found!"

So three documented "sharp edges" are deleted rather than reworded: ** silently degrading with globstar off (which was silently dropping every top-level task), a non-matching glob reaching the CLI as a literal path, and the need for an explicit per-depth ladder.

The other correctness win is the one that motivated args in the first place, reproduced with one touch:

$ touch 'sandbox.docker.env_passthrough_extra=A'
# whitespace-split (the old extra-args):
  [-D] [sandbox.docker.env_passthrough_extra=A]
# args:
  [-D] [sandbox.docker.env_passthrough_extra=[AUTH_TOKEN,BASE_URL]]

[...] is a bash character class. Silent, and it changes what the run measures. That path no longer exists.

The one capability this removes

minimum-task-score and its ~50 lines of embedded Python are gone with no replacement. Both call sites set it to "0.0", a no-op, so it had no real user. It is also in the wrong place: a score floor is policy over run.json, so it belongs in the CLI as --min-task-score, where it is unit-testable and reachable from the ADO pipelines and the nightly VM, which cannot use a GitHub action at all and reimplement their own gating today. Small follow-up, deliberately not in this PR.

Evalboard side

A third source, gha -> container runs-gha, registered and deliberately unlisted: no tab, no listing page, no aggregate view. Registration and surfacing turn out to be independent (NAV is a hardcoded array that does not iterate SOURCES), and app/runs/[id] reads by id on demand, so a direct link resolves with no listing in existence. That is what lets this skip the one piece of evalboard work with a documented footgun.

Its own container rather than a prefix inside runs, because getAdhocRunListing loads per-run metadata for every non-date-shaped id before truncating to the display limit, and because the storage account now carries a 14-day expiry rule scoped to runs-gha/ that must never be able to reach nightly history.

Three stale surfaces found while trimming the comments

Each one described an input that no longer exists, and each was invisible to the pinning lint:

• The ci skill still told the agent to pass an experiment via extra-args:, so an emitted workflow would have had its experiment silently ignored — exactly the forwarding-input failure this PR is about, aimed at the workflows the plugin writes. Now -e and the path as two args lines.
• docs/CI_GATE.md carried "Extras and plugins" twice; the second copy predated the rewrite and still named prerelease. Its frontmatter and intro also still promised a per-task score floor and a job-summary write.
• test_ci_skill_does_not_recommend_a_recursive_task_glob enforced the opposite of the new behaviour: it banned ** and required the skill to explain globstar. It was passing only because the rewritten skill contains the phrase "no globstar caveat". Removed — CE026's unknown-input check already covers a stale tasks: in a snippet.

After merging

Two dispatches, in this order, before the skills PR can do anything:

1. Release (release.yml), which bumps the version, re-pins action.yml, and moves v0. Until v0 moves, a consumer passing these inputs gets them silently ignored. Note the reverse window too: between this merge and that release, verify-published-action.yml on main passes the new with: block to the old @v0, so its nightly runs the same single task on the default model instead of haiku. Cheap, not broken, but a reason to release promptly.
2. deploy-evalboard.yml in coder_eval_uipath (manual dispatch, no post-deploy health check). The gha source only exists on the deployed site once that runs, so sequence it before the first upload or the emitted link 404s on arrival.

Verification

• tests/test_action_inputs.py (37 cases) executes both step scripts pulled out of action.yml with uv/coder-eval stubbed to record argv, so it asserts the text that ships rather than a copy. Mutation-checked: flattening the junit derivation fails 5 of them.
• The stub is written in bash, not Python, because Git Bash rewrites absolute-POSIX-looking arguments on the way to a native binary, so a python-shebang stub mangled /action-checkout on the Windows runner.
• CE026, the repo's own lint rule for doc snippets passing inputs action.yml does not declare, caught a surface I had missed (plugins/coder-eval/skills/ci/SKILL.md). Worth noting the rule earned its keep.
• action-dogfood covers what a unit test cannot: working-directory on a composite step, a plugin installed through extra-packages being discovered at runtime, and now the run-md-path summary-append recipe the docs tell consumers to use. The discovery probe is a coder-eval plan on a task naming the BYOA fixture's agent kind, which exits 1 with No agent registered for type 'byoa-demo' if the entry point was not found.
• Full local sweep: 426 lint/action tests, 608 evalboard vitest cases, tsc --noEmit, next build, ruff and pyright clean. actionlint findings unchanged against baseline on every workflow touched.

🤖 Generated with Claude Code