diff --git a/README.md b/README.md index 0b50976..2b838e0 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,8 @@ tailnet ## [Lima](docker-arm) +> **Superseded:** the `aarch64-linux` VM now runs under Tart, not Lima, so it can share a vmnet bridge with the macOS VM — see [Fast VM-to-VM networking](#fast-vm-to-vm-networking-tart--softnet). These Lima steps are kept for reference. + Similarly, the ARM Linux config can be used for a Linux virtual machine on macOS, via [Lima](https://lima-vm.io/) which comes with the host-side macOS config in this repo. First create the VM: - The username and home directory location must be set to match what this Home Manager config expects. @@ -367,4 +369,26 @@ sudo tailscale up --ssh --hostname=tahoe-vanilla Note that, without additional setup, this VM can only receive Tailscale SSH connections, and cannot SSH into other VMs on the tailnet. +## Fast VM-to-VM networking (Tart + softnet) + +Both sandbox VMs are on a [Tailscale](https://tailscale.com/) tailnet, but two VMs on the **same** Mac can't reach each other directly, so Tailscale relays their traffic through a [DERP](https://tailscale.com/kb/1232/derp-servers) server instead of taking the local path. The cause: Lima (`vzNAT`) and Tart (default) each put their guest behind Apple's per-VM NAT ([`VZNATNetworkDeviceAttachment`](https://developer.apple.com/documentation/virtualization/vznatnetworkdeviceattachment)), which turns on vmnet **bridge isolation** and blocks cross-VM traffic. There is no way to clear that isolation on the shared `bridge100`, and each tool's _non_-isolating mode (Lima's `socket_vmnet`, Tart's `softnet`) builds its own separate bridge — so a Lima guest and a Tart guest can never share one. + +The fix is to run **both** VMs under **Tart**, each on [softnet](https://github.com/cirruslabs/softnet) with isolation disabled: + +```sh +tart run --no-graphics --net-softnet --net-softnet-allow=0.0.0.0/0 +``` + +`--net-softnet-allow=0.0.0.0/0` disables softnet's bridge isolation, so both VMs land on one shared vmnet (`192.168.2.0/24`) and reach each other directly — so Tailscale connects them directly instead of relaying. Verified: two softnet VMs land on `192.168.2.2`/`192.168.2.3` and ping each other at ~0.5 ms. This means the `aarch64-linux` VM moves from Lima to Tart — create it with `tart clone ghcr.io/cirruslabs/ubuntu:latest sandbox-arm64` and provision it with the [docker-arm](docker-arm) Home Manager config, exactly as the Lima VM was. + +Both VMs are autostarted by the LaunchAgents in [`macos/home-manager/home.nix`](macos/home-manager/home.nix). Two things Home Manager can't do for you: + +1. **softnet must be setuid root** to drive vmnet. After `home-manager switch` puts the `softnet` package on `PATH`, do the one-time privileged install (repeat whenever the `softnet` version changes): + + ```sh + sudo install -o root -g wheel -m 4755 "$(command -v softnet)" /usr/local/bin/softnet + ``` + +2. **The VMs start at login, not at boot.** A Virtualization.framework VM needs a user GUI session, so a boot-time `LaunchDaemon` can't run one — enable auto-login for a headless restart to bring the VMs back. (Tart + launchd autostart is new here and likely needs iteration; some of the startup/shutdown pain is macOS-intrinsic — the same session and DHCP-lease issues Lima had.) + [flakes]: https://wiki.nixos.org/wiki/Flakes#Other_Distros,_without_Home-Manager diff --git a/macos/home-manager/home.nix b/macos/home-manager/home.nix index 6bf877a..4bbd015 100644 --- a/macos/home-manager/home.nix +++ b/macos/home-manager/home.nix @@ -20,6 +20,7 @@ pkgs.gh pkgs.lima pkgs.tart + pkgs.softnet ]; # Necessary for `git send-email` to work. @@ -61,5 +62,47 @@ }; }; + # Autostart both sandbox VMs under Tart, each on softnet with isolation + # disabled (`--net-softnet-allow=0.0.0.0/0`). That drops them onto one shared + # vmnet bridge (192.168.2.0/24) where they can reach each other directly, which + # in turn lets Tailscale punch a direct connection between them instead of + # relaying — so two VMs on this one Mac stop hairpinning their traffic out to + # the NixOS DERP relay and back (see the README's networking section). + # + # These are LaunchAgents, so they start at *login*: a Virtualization.framework + # VM needs the user's GUI session, so a boot-time LaunchDaemon can't run one — + # enable auto-login for a headless restart. `tart run` blocks while the VM is + # up, and KeepAlive brings it back if it exits. + # + # softnet must be setuid root (it drives vmnet), which Home Manager can't set, + # so it's a one-time privileged install (see README). The PATH below points + # Tart at that setuid copy in /usr/local/bin. + launchd.agents = + let + tartVM = name: { + enable = true; + config = { + ProgramArguments = [ + "${pkgs.tart}/bin/tart" + "run" + "--no-graphics" + "--net-softnet" + "--net-softnet-allow=0.0.0.0/0" + name + ]; + RunAtLoad = true; + KeepAlive = true; + EnvironmentVariables.PATH = "/usr/local/bin:/usr/bin:/bin"; + StandardOutPath = "/Users/samueles/Library/Logs/tart-${name}.log"; + StandardErrorPath = "/Users/samueles/Library/Logs/tart-${name}.log"; + }; + }; + in + { + # The aarch64-linux VM, migrated off Lima; the aarch64-darwin VM. + tart-sandbox-arm64 = tartVM "sandbox-arm64"; + tart-tahoe-vanilla = tartVM "tahoe-vanilla"; + }; + programs.zsh.enable = true; # Necessary for aliases and Starship to work. }