Sandboxed tenants: an account gets its own machine, not a shell on ours - #56
Merged
Conversation
…ne, not a shell
Selling access to a box currently means handing out a normal UNIX account,
which is only as strong as every world-readable file, every localhost
service and every kernel LPE published before the next reboot. That is fine
for colleagues and wrong for customers.
A tenant now gets an incus instance instead: an unprivileged container by
default, or a real KVM VM (--sandbox=vm) for the "I need root" tier. Their
host account exists only to own /home/<login> -- nologin shell, never in
sudo or admin -- and that home is idmapped into the instance, so
~/public_html and ~/apps keep working through the existing nginx with no
change to the vhosts.
./root-ubuntu.sh alice --sandbox
./root-ubuntu.sh bob --sandbox=vm
./root-ubuntu.sh sandbox {init,create,refresh,enter,exec,start,stop,rm}
What keeps a tenant off everything else:
* no host-side execution at all -- the curl|sh tool installers (oh-my-zsh,
mise, moshcode) and oh-my-tmux run INSIDE the instance for a tenant.
Running them out here reads their own dotfiles as their own uid, which
is host code execution for anyone who edits ~/.gitconfig
* a network ACL rejecting RFC1918, 169.254 (cloud metadata) and the
tailnet, which also stops tenant reaching tenant
* caps on cpu/memory/processes, and disk when the pool driver can enforce
one (btrfs/zfs; dir cannot, and says so)
* one forwarded ssh port each, derived from a slot allocated once and
never moved
Dev apps keep working because each tenant owns a band of loopback ports
forwarded into their instance, so nginx's proxy_pass to 127.0.0.1 is
unchanged. profullstack-devapps now refuses a .port outside the writer's
own band -- otherwise alice could publish bob's running service under her
own name. Bands are enforced only where they are ALLOCATED, so an existing
account already listening on a port in the region is not broken by turning
this on.
Idempotent throughout: pools, bridges, ACLs, profiles and devices are
compared before they are written, the in-instance bootstrap is gated on a
revision marker, and a re-run reconciles rather than recreates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y32FRASN3UZhUgwFdbEAzb
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ZSH="$HOME/.oh-my-zsh" ZDOTDIR="$HOME" RUNZSH=no CHSH=no KEEP_ZSHRC=yes \ | ||
| sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" >/dev/null 2>&1 | ||
| curl -fsSL https://mise.run | sh >/dev/null 2>&1 | ||
| curl -fsSL https://moshcode.sh/install.sh | sh >/dev/null 2>&1 |
ThreatCrush Security Scan12 finding(s) HIGH/CRITICAL: 4 | MEDIUM: 4 | LOW: 4
Snippets are redacted; ThreatCrush never prints matched credential material. |
ralyodio
added a commit
that referenced
this pull request
Sep 6, 2026
master merged #56, which puts a tenant in their own incus container or VM under the name `sandbox`. This branch confines accounts in place. They are layers rather than rivals -- a tenant with an instance is still a non-admin account out here -- so both survive the merge intact. The rename in the previous commit did most of the work; what was left: * both features had inserted a section at the same point, and git had folded the shared "esac }" tail of cmd_confine and cmd_sandbox together. cmd_confine's case is closed explicitly and both sections are kept * one subcommand list, one dispatch arm each * a comment about ssh-agent lingering that #56 orphaned above the sandbox stage is back with the stage it describes Checked rather than assumed, since git auto-merged several regions both branches had touched: * fix_home_permissions keeps BOTH sides -- the _confine_home_mode branch and the per-app publishing walk * add_user_spec keeps both the tenant-groups shortcut and ask_groups * DEFAULT_GROUPS still moves to `users` under confinement * stage order is right: incus up, accounts created, THEN confinement, which is what makes group membership the thing that decides a tier Tenants are excluded from _confine_humans (nologin), which is correct: they have no host processes to cap. Their home mode is still set, because that runs per-home out of fix_home_permissions rather than off that list. Three defects fixed on the way through: * the accounts table hand-padded "admin" to the width of "sandboxed", so the rename to the shorter "confined" misaligned the column. It pads in the format string now * _confine_sysctl_floor's `want` local collided with an array of the same name elsewhere; the linter does not scope locals, so it cost two false warnings. Renamed to `floor` * two tests pinned the whole subcommand alternation and broke when `sandbox` joined it. They match the verb inside the list now, so the next subcommand does not break them 791 tests pass; shellcheck clean apart from one SC2088 that predates both branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y32FRASN3UZhUgwFdbEAzb
This was referenced Sep 6, 2026
ralyodio
added a commit
that referenced
this pull request
Sep 6, 2026
`/cli-tools crawlproof update` printed the dashboard's usage and exited 2. The wrapper spells its own flags `--self-*` so that every plain word belongs to upstream, which is the right default and was the wrong answer here: `update` is the word `cli-tools update` already uses in this repo, so getting usage back is the command being wrong rather than the person. So the wrapper claims exactly three words -- update, upgrade, self-update -- and only in first position, which keeps `crawlproof ads budget update` upstream's. Installing is the wrapper's job and can never come to mean something in the dashboard, so this is the one place the rule is safe to break. The predicate lives in src/ and is exported, so the tests exercise the real thing rather than a copy of the word list. Note: test/root-ubuntu.test.ts fails on master as of #56, unrelated to this. Claude-Session: https://claude.ai/code/session_01HvWJ4336pxTFRdRbvsTQeD Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Sep 6, 2026
`crawlproof update` said "installed with pnpm" and left the old version in place, so the bug it was run to fix was still there. Same shape for `hqtui`. pnpm 11 ships a `minimumReleaseAge` cooldown that refuses versions published in the last little while, and it does not fail when it refuses one: it resolves to the newest release old enough to pass, writes a note about an exclude list, and exits 0. Reproduced in an empty directory on pnpm 11.18.0, with the registry reporting 0.2.0: pnpm add @profullstack/crawlproof@latest -> 0.1.0, exit 0 npm install @profullstack/crawlproof@latest -> 0.2.0 So both wrappers now ask what landed instead of trusting the exit code, and move to the next package manager when the answer is the wrong version. The version installed is printed, because "installed" without a number is exactly the claim that turned out to be false. Deliberately not `--config.minimumReleaseAge=0`. The cooldown is a real supply-chain protection, and switching it off wholesale in a tool that installs on other people's machines is a bigger decision than fixing an update. Falling through to npm leaves it as pnpm's default and still lets a deliberate update finish. An unreachable registry means the wanted version is unknown, and an unknown want passes: an offline box must still be able to reinstall what it has. Two failures on this branch are not from it: root-ubuntu's `groups` test fails on master as of #56, and registry's summary test fails on an untracked bin/argontv.ts sitting in the working tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvWJ4336pxTFRdRbvsTQeD
ralyodio
added a commit
that referenced
this pull request
Sep 6, 2026
`crawlproof update` said "installed with pnpm" and left the old version in place, so the bug it was run to fix was still there. Same shape for `hqtui`. pnpm 11 ships a `minimumReleaseAge` cooldown that refuses versions published in the last little while, and it does not fail when it refuses one: it resolves to the newest release old enough to pass, writes a note about an exclude list, and exits 0. Reproduced in an empty directory on pnpm 11.18.0, with the registry reporting 0.2.0: pnpm add @profullstack/crawlproof@latest -> 0.1.0, exit 0 npm install @profullstack/crawlproof@latest -> 0.2.0 So both wrappers now ask what landed instead of trusting the exit code, and move to the next package manager when the answer is the wrong version. The version installed is printed, because "installed" without a number is exactly the claim that turned out to be false. Deliberately not `--config.minimumReleaseAge=0`. The cooldown is a real supply-chain protection, and switching it off wholesale in a tool that installs on other people's machines is a bigger decision than fixing an update. Falling through to npm leaves it as pnpm's default and still lets a deliberate update finish. An unreachable registry means the wanted version is unknown, and an unknown want passes: an offline box must still be able to reinstall what it has. Two failures on this branch are not from it: root-ubuntu's `groups` test fails on master as of #56, and registry's summary test fails on an untracked bin/argontv.ts sitting in the working tree. Claude-Session: https://claude.ai/code/session_01HvWJ4336pxTFRdRbvsTQeD Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
"Resell a VPS if they want root, otherwise put them somewhere they can't break other people's stuff."
The name for the thing you were describing is a container — specifically an unprivileged system container, which is a whole machine (own init, own package manager, own users) sharing this box's kernel. "Sandbox" normally means something much smaller (seccomp/bubblewrap around one process) and is the wrong model here. The paid tier is a VM (own kernel).
The FOSS manager is incus — the LXD fork the original LXC/LXD maintainers moved to. Apache-2.0, in Ubuntu 24.04's own archive, one daemon, one CLI, and the same CLI drives both containers and VMs — which is why both tiers here are ~20 lines apart instead of two implementations.
Usage
--refreshreconciles every existing tenant.--skip-sandboxleaves them alone.What actually stops a tenant reaching anything of ours
nologin, password locked, never insudo/admincurl | shinstallers (oh-my-zsh, mise, moshcode) and oh-my-tmux run inside the instance for a tenant. Running them on the host reads their dotfiles as their uid — that is host code execution for anyone who edits their own~/.gitconfig, and it was the sharpest edge in the whole change169.254.169.254(cloud metadata) and the100.64/10tailnet are rejected. That also stops tenant→tenant, since a neighbour is only ever an address inside our own bridgedircannot, and the script says so out loud rather than accepting a quota it will ignore)2200 + slot, forwarded to the instance's own sshd. The host's sshd is never told about themHow their work still shows up on the web
/home/<login>is bind-mounted into the instance (idmapped, so uids match on both sides). That single device is what keeps the rest of the script working untouched —~/public_htmlis still served by the host's nginx,~/apps/<app>/.portis still compiled into the devapps map, and no vhost changed.Each tenant owns a band of loopback ports (slot 0 →
21000-21099) forwarded127.0.0.1:N ↔ 127.0.0.1:Ninto their instance, so nginx's existingproxy_pass http://127.0.0.1:$devapp_portreaches it with no idea anything moved.profullstack-devappsnow refuses a.portoutside the writer's own band. Without that, alice could put bob's port in her own~/apps/x/.portand serve his unreleased work under her name. Bands are enforced only where allocated — an account already listening on a port in the region is not broken by turning this on.Idempotency
Pools, bridges, ACLs, profiles and devices are compared before they are written (YAML documents against the checksum of what we last sent, since incus echoes back defaults and reorders maps). The in-instance bootstrap is gated on a revision marker. A slot, once allocated, is never moved — the ports are in DNS, in people's
~/.ssh/configand in nginx. A re-run reconciles; it never recreates.Testing
Nothing here was run against a live incus daemon — this needs a real box.
bash -nclean;shellcheck -S warningclean apart from one pre-existing SC2088 at line 998container→vm, no duplicate row), free, reallocate into the gapprofullstack-devappswas extracted, parsed and run against fake homes: tenant publishes own band ✅, tenant reaching a neighbour's band ✅ blocked, non-tenant squatting an allocated band ✅ blocked, non-tenant on an unallocated port in the region ✅ still allowed, a box with no tenants ✅ behaves exactly as beforesandbox --help, baresandbox,--sandbox=<bad kind>, non-rootsandbox rm, andgroups --helpall check outOne bug the tests caught and the fix kept:
_tenant_registeroriginally printed its slot on stdout while also callingnote(), so a caller reading it through$(...)spliced the log line into the slot number and lost theCHANGEDentry to the subshell. It sets$TENANT_SLOTnow.Before this runs on dev.chovy.com
incus network aclneeds the nftables firewall driver. If ACL creation fails the script warns and continues — the tenant would then have no egress isolation, so check that warning on the first run.*.<user>.dev.chovy.com(unchanged from today, but now it matters per customer).SANDBOX_MEMORYdefaults to 2GiB each. Ten tenants is 20GiB of cap; check the box against [fleet OOM hardening] before selling the tenth.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y32FRASN3UZhUgwFdbEAzb