Skip to content

fix(orch): keep root-executed helpers unwritable under the world-writable /usr/local - #3439

Closed
tomassrnka wants to merge 2 commits into
mainfrom
fix/usr-local-sticky
Closed

tomassrnka wants to merge 2 commits into
mainfrom
fix/usr-local-sticky

Conversation

@tomassrnka

@tomassrnka tomassrnka commented Jul 29, 2026 •

Copy link
Copy Markdown
Member

finalize/configure.sh runs chmod -R 777 /usr/local so sandboxes can install there without sudo, which also left the helpers root executes at boot (e2b-seed-certs, envd.openrc, ssl-certs.tar, distro.env) replaceable by any user. Finalize now sticky-bits /usr/local and the three subdirectories holding them and restores root-owned 755/644 on the files — sticky keeps sudo-less installs working while limiting unlink/rename to the owner. The whole ancestor chain needs it: a sticky directory under a non-sticky world-writable parent can be renamed aside wholesale (thanks Bugbot).

Defense in depth, not an escalation fix — the sandbox user has NOPASSWD: ALL by design; this stops root from executing files anything in the sandbox could have swapped. Verified on real KVM per family plus a fromTemplate child: envd's ExecStartPre still exits 0 (including a live envd restart), and as user an install into /usr/local/bin succeeds while replacing e2b-seed-certs fails.

@cla-bot cla-bot Bot added the cla-signed label Jul 29, 2026
@cursor

cursor Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Guest image finalize script and cert-pack step only; no runtime API or auth changes.

Overview
World-writable /usr/local from finalize let any sandbox user replace root-run boot helpers (e2b-seed-certs, provision runner, envd.openrc, distro.env). Finalize now applies sticky bits along /usr/local through share/e2b, resets those helpers to root-owned non-writable modes, and after packing ssl-certs.tar sets root-owned 644 so a prior chmod -R 777 or in-place rebuild cannot leave it user-writable.

Reviewed by Cursor Bugbot for commit 42b5793. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
3721 1 3720 9
View the top 1 failed test(s) by shortest run time
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxObjectNotFound
Stack Traces | 181s run time
=== RUN   TestSandboxObjectNotFound
=== PAUSE TestSandboxObjectNotFound
=== CONT  TestSandboxObjectNotFound
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox ilui9xriub4537s2ptvpf (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox ie1ye79pzzxy7oxwego79 (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox i766guui2413lrz33x7xa (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox i1kp6uva0oarkda7hsokm (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox i766guui2413lrz33x7xa (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox ij1f0nftdte0xcnzqq3qv (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox iw86svwk9y40o5vd88gxc (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
Executing command bash in sandbox iw86svwk9y40o5vd88gxc (user: root)
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:62: sandbox creation failed due to resource exhaustion, retrying
    sandbox_object_not_found_test.go:73: failed to create sandbox after 36 retries
--- FAIL: TestSandboxObjectNotFound (181.29s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@tomassrnka
tomassrnka force-pushed the fix/usr-local-sticky branch from 22894ce to fd3a1ef Compare July 29, 2026 09:51
@tomassrnka

Copy link
Copy Markdown
Member Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit fd3a1ef. Configure here.

@tomassrnka
tomassrnka marked this pull request as ready for review July 29, 2026 10:14

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't find any additional bugs, but this PR changes the permission and ownership model for root-executed boot helpers (envd's ExecStartPre chain) across every sandbox template's finalize phase, so it's worth a human look given the security-sensitive nature of the change and the fact that Bugbot already caught and required a fix for an ancestor-chain sticky-bit gap during this same review cycle.

What was reviewed: the sticky-bit/ownership restoration logic in configure.sh covering the full /usr/local ancestor chain and the helper binaries, and the ssl-certs.tar hardening added in configure.go. Candidate issues examined and ruled out this run: symlink-following in the chown/chmod restore loops, a pre-planted symlink at the ssl-certs.tar path being followed by tar/chown/chmod, and the sticky bit blocking legitimate sudo-less reinstall/removal of files in /usr/local/bin.

Extended reasoning...

Overview

This PR modifies packages/orchestrator/pkg/template/build/phases/finalize/configure.sh and configure.go, the guest-side script run as root during template finalize. It adds sticky bits to /usr/local and its subdirectories (which are chmod -R 777'd earlier in the same script to allow sudo-less installs) and restores root ownership/non-writable modes on the helper files that envd executes as root at boot (e2b-seed-certs, e2b-provision-runner, envd.openrc, distro.env, ssl-certs.tar). The intent is defense-in-depth: prevent a sandbox user (who already has NOPASSWD sudo) from swapping the content of files root executes at boot, without breaking sudo-less installs into /usr/local.

Security risks

This is inherently a file-permission hardening change to a boot-time root-execution path that runs inside every sandbox template. The main risk categories are: (1) incompleteness — missing a directory or file in the ancestor chain that still allows a swap (this already happened once during review: Bugbot found that sticky bits on only the leaf directories could be bypassed by renaming a non-sticky parent, and the author fixed it by sticky-bitting the whole chain with KVM verification across four distro families), and (2) regressions to legitimate sudo-less installs into /usr/local (verified by the author not to be broken). No injection, auth-bypass, or data-exposure vectors are introduced; this is a hardening measure on top of an already-privileged execution context (the sandboxed user already has full root via sudo).

Level of scrutiny

Permission and ownership handling for root-executed code is squarely in the security-sensitive category, and it applies to the shared finalize path for all templates, so I think a human maintainer should confirm the approach even though the bug-hunting system found nothing further and the author already responded thoroughly to Bugbot's finding with real hardware verification across ubuntu/fedora/arch/alpine plus a fromTemplate rebuild.

Other factors

The author's own KVM testing (before/after probes showing the swap succeeding then failing, and ExecStartPre still exiting 0) is strong practical validation, but the failing integration tests reported by Codecov on this PR's earlier commit (TestDeleteTemplate, TestTemplateBuildFuseConfiguration, TestCommandKillNextApp, all failing at the finalize step with a generic internal error) are unresolved in the timeline. Given TestTemplateBuildRUN shows a known ~13-18% flake rate on main, these may be pre-existing CI flakiness rather than caused by this change, but that hasn't been confirmed either way, which is another reason for a human to check CI status before merging.

@tomassrnka
tomassrnka force-pushed the fix/usr-local-sticky branch from fd3a1ef to e9db1cb Compare July 29, 2026 15:38
@tomassrnka tomassrnka closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant