Skip to content

feat(sandbox): CI capabilities benchmark for Actions-product readiness - #503

Closed
kisernl wants to merge 1 commit into
masterfrom
devin/1791823678-ci-capabilities-bench
Closed

kisernl wants to merge 1 commit into
masterfrom
devin/1791823678-ci-capabilities-bench

Conversation

@kisernl

@kisernl kisernl commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds benchmarks/sandbox/ci-capabilities.bench.ts — a real-fidelity gate for Actions-product readiness. One iteration creates a sandbox, runs the full capability battery, and destroys it; a provider that passes every check can be dropped into the benchmarks-platform CI executor as-is. Check length is deliberately not a concern — the goal is fidelity, not speed.

The heavy tail of the battery re-runs the same scripts the product's executor generates, ported verbatim from lib/ci/executor/act-runtime.ts into benchmarks/sandbox/ci-runtime.ts:

  • dockerd — real daemon bring-up (static Docker 27.4.1 install, nohup setsid launch, socket poll, per-provider flags: blaxel's second daemon on unix:///var/run/docker-ce.sock + ci-docker0, cloudflare's --iptables=false --ip6tables=false --ip-forward=false)
  • act_install — the pinned act v0.2.89 fetch/extract/sha-marker script
  • runner_image — catthehacker/ubuntu:act-* pull (or the blaxel in-box runner image build + --pull=false)
  • act_run — a detached act push running a real workflow to a green exit, with bridge-gateway artifact-server resolution, --bind, --action-offline-mode, and the platform's -P image args

The earlier checks cover the lifecycle contract: exec semantics (env injection, exit-code/stdout-stderr separation, per-command kill), getById re-attach, concurrent exec, detached-process survival, writable workspace, Node runtime (or fetch-tool bootstrap path), per-host egress reachability (github/codeload/npm/docker/platform/nodejs), a real git clone --depth 1, and the 4 vCPU / 8 GiB / 16 GiB resource floor. deadline creates a second sandbox with a 4-minute timeout and verifies the box actually dies on schedule (catches auto-pause and ignored-TTL bugs like the namespace 1h issue); it runs even when the main sandbox fails to create.

Each check runs inside its own step() and records {passed, error} — one failure never aborts the battery, so a provider's exact gap shows up in record.data.features and results/sandbox-ci-capabilities/latest.json. Scoring is checksPassedPct.

Supporting changes:

  • resource-options.ts — the LARGE_RESOURCE_OPTIONS sizing map extracted from dax.bench.ts so this bench and dax share the 8 vCPU / 16 GiB per-provider sizing
  • sandbox-capabilities.yml — repointed at the new bench (the old capabilities.bench.ts was deleted earlier), job timeout raised to 120min, results dir/artifact names renamed to ci-capabilities
  • package.json — bench:sandbox:ci-capabilities script

Verified: pnpm typecheck clean; bench loads and config-validates under --dry-run. A live provider run needs vault keys, so the first real signal comes from dispatching the workflow.

Link to Devin session: https://app.devin.ai/sessions/d9e314eb6c40479fbefd4d5bb85edc77
Open in Devin Desktop: https://app.devin.ai/desktop/session/d9e314eb6c40479fbefd4d5bb85edc77?variant=devin
Requested by: @kisernl

Adds benchmarks/sandbox/ci-capabilities.bench.ts: a real-fidelity gate that
replays the benchmarks-platform CI executor path inside one sandbox —
exec semantics, re-attach, concurrent + detached exec, workspace/runtime,
egress reachability, git clone, resource floor, verbatim dockerd bring-up
and act install scripts (ci-runtime.ts, ported from lib/ci/executor/
act-runtime.ts), runner-image pull, a detached act workflow run to green,
and a second sandbox verifying the instance deadline is honored.

Each check reports {passed, error} into a capability matrix in record.data
and results/sandbox-ci-capabilities/latest.json so a provider's readiness
(or exact gap) is visible at a glance. A provider passing every check can
be dropped into the Actions product.

Also extracts the shared LARGE_RESOURCE_OPTIONS sizing map from
dax.bench.ts into resource-options.ts and repoints the
sandbox-capabilities workflow at the new bench (120min job timeout).

Co-Authored-By: Noah Kiser <noah@computesdk.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Closing — this moved to the standalone repo computesdk/benchmarks-sandbox-ci-capabilities (initial commit on main), following the benchmarks-sandbox-fs-api standalone-repo pattern for providers/env/vault wiring. The dax resource-options.ts extraction was only needed by this bench, so nothing from this PR is needed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant