feat(snapshot): add full checkpoint capture and restore - #1503
feat(snapshot): add full checkpoint capture and restore#1503appcypher wants to merge 30 commits into
Conversation
Validate complete checkpoint closures and decode execution and device envelopes before VM construction. Stream verified memory objects once into the inert guest, preload all captured state before first activation, and carry an optional child-owned restore source through the launch contract.\n\nThe public snapshot restore path remains disabled until the workload latch and VM generation activation gate are connected.
Add generation-8 workload freeze and thaw messages and place every agentd-launched process into a cgroup-v2 freezer before exec. Keep normal execution available when the freezer is absent while reporting resumable capture as unavailable. Hold the exact attempt-scoped latch across VM pause, component capture, and root-last publication. Resume and thaw in order, and fail closed when either transition cannot be established.
Persist the captured agent protocol identity in checkpoint resources and restore it through a private console exchange. Resume restored guests only after VM Generation ID processing, release the attempt-scoped workload latch, and synthesize the cached ready handshake. Keep agent and control endpoints unpublished until activation succeeds so clients cannot reach a partially restored runtime.
Add a one-shot network start gate that is selected only for checkpoint restore. Hold smoltcp processing and published-port listener creation until the restored guest has crossed VMGenID processing, workload thaw, and local endpoint publication. Ordinary boots remain immediately active, and the gate is consumed before the poll loop so packet processing has no steady-state check.
Use the existing runtime control endpoint to publish a validated composite checkpoint into a schema-1 snapshot artifact. Materialize immutable checkpoint members without copying runtime-private state and preserve the old artifact until force replacement succeeds.
Materialize installed resumable snapshots into sandbox-owned checkpoint and disk state before construction. Restore captured VM geometry and guest network identity while creating a fresh private qcow2 head for the child.
Capture running checkpoints directly into archives, include exact checkpoint closures when saving installed snapshots, and materialize archives directly into child-owned restore state. Support bounded GNU long-name records for content-addressed closure paths and preserve explicit restore constraints until deferred archive resolution.
Extend explicit snapshot verification to checkpoint-backed artifacts while preserving the released disk verification projection. Stream every referenced memory object, report the verified checkpoint root across the CLI and SDKs, and cover corrupt-memory detection.
Allow checkpoint integrity verification and archive transport across host architectures while keeping construction-time restore admission architecture-specific.
Update the CLI, Rust, Python, TypeScript, and Go snapshot guidance for running checkpoint capture, direct archives, eager child restore, and full checkpoint verification.
Rename the public running-checkpoint mode to full across the CLI and SDKs while retaining structural checkpoint descriptor values and legacy index decoding. Add filesystem-clean capture preparation, disk-only cold restore for installed and direct-archive checkpoints, resume-only full restore command handling, and synchronized user-facing and planning documentation.
Update stale full-capture and GNU long-name assertions, and isolate the sparse archive test from host database state.
Skip ordinary OCI root-disk defaults and growth for installed checkpoint restores so full and disk-only children retain captured disk geometry without mutating hard-linked sealed layers.
Use the representation emitted by the archive writer as the source of truth for encoded size, apparent size, sparse ranges, and transport integrity. This keeps inventories valid when sparse encoding falls back to a dense GNU long-name member and avoids stale pre-scan metadata. Update aggregate limits in constant time with checked replacement, and cover the 101-byte checkpoint qcow2 member path with a direct archive round trip.
Reconstruct runtime-owned passthrough and single-file filesystem state from destination-local paths, including inode identities, open handles, stable directory snapshots, writeback state, and quota accounting.\n\nRestore bootstrap namespaces before device activation, require a ready VM generation driver for full capture, and force resumed workloads onto detached lifecycle ownership before spawning the runtime.
Keep Python missing-snapshot validation synchronous while deferring descriptor parsing and full/disk admission to the shared Rust resolver. Regenerate the Node native bindings so direct archive results are exported, and cover the required native constructor in the contract suite.
Treat an empty round-tripped network interface as defaults rather than an explicit identity override, while still rejecting populated conflicts. Import the Windows access-mode mask required to rebuild captured file handles on native Windows targets.
Make the captured file-handle collection explicit so native Windows can type-check its validation before serializing passthrough state.
Keep the installed checkpoint source separate from a direct archive restore that has already populated child-owned staging. This avoids copying the eager restore closure onto itself and losing the extracted artifact.
Re-evaluate local process ownership after a direct archive descriptor is materialized. Full restores now avoid parent watchdogs and Windows jobs even when their scope was unknown during outer builder dispatch.
|
| if let Some(checkpoint) = response.checkpoint { | ||
| if !response.ok { | ||
| tracing::warn!( | ||
| sandbox = name, | ||
| error = response.error.as_deref().unwrap_or("source resume failed"), | ||
| "checkpoint published but the source runtime did not return to running" | ||
| ); | ||
| } | ||
| return Ok(checkpoint); |
There was a problem hiding this comment.
Quiesced source reported successful
When VM resume or workload thaw fails after checkpoint publication, this branch returns Ok(checkpoint) even though the runtime has transitioned to Quiesced. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.
Knowledge Base Used: Runtime execution orchestration
Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk/rust/lib/sandbox/modify.rs
Line: 802-810
Comment:
**Quiesced source reported successful**
When VM resume or workload thaw fails after checkpoint publication, this branch returns `Ok(checkpoint)` even though the runtime has transitioned to `Quiesced`. The source remains reported as running but cannot execute normally or accept later checkpoint, CPU, memory, or secret mutations, and the control API provides no operation to resume or thaw it.
**Knowledge Base Used:** [Runtime execution orchestration](https://app.greptile.com/microsandbox/-/custom-context/knowledge-base/superradcompany/microsandbox/-/docs/runtime-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Record monotonic phase timings for stopped and full snapshot creation, archive handling, checkpoint capture, memory restore, and activation. Emit capture telemetry only after the source workload is resumed and thawed so benchmark logging does not extend the measured pause window.
Keep zero detection at 2 MiB while packing non-zero sparse extents into bounded 32 MiB immutable objects. This amortizes hashing, durable object publication, directory work, and restore object reads without changing logical memory coverage.
Do not compare a selected platform manifest body with the parent OCI index digest recorded by multi-platform pulls. Continue binding metadata to the snapshot digest and validating the image config and layer identities.
Persist independent device envelopes and link immutable memory objects with bounded parallel workers after capture. Preserve inventory ordering and existing object verification while reducing time spent in the paused epoch.
Gate the guest heartbeat writer during workload freeze and remove any abandoned temporary heartbeat before acknowledging the checkpoint barrier. Resume writes on thaw or rollback so captured passthrough state never references a transient path absent at restore.
Record the runtime-owned root layout in schema-1 descriptors and preserve flat roots through stopped and full capture, installed and direct archives, disk-only restore, and eager resume. Keep earlier descriptors compatible by defaulting the missing layout to managed. Share checkpoint rollover and restart recovery across managed /dev/vdb uppers and flat /dev/vda roots. Preserve complete raw/qcow2 chains after rollover so later disk snapshots retain post-checkpoint writes. Materialize patched flat roots privately, carry tmpfs state only through full memory snapshots, and retain cloud and user-owned disk-image rejection. Refuse chain-backed growth until the active qcow2 head can be resized without mutating a sealed ancestor.
|
Too many files changed for review (135 files, 100 file limit). Bypass the limit by tagging |
Keep standalone raw disks on the bounded-writeback path. Relocate qcow2 backing names only in private copies and hash the published bytes; stream rewritten headers for direct archives without staging a disk. Delay artifact staging until runtime capture succeeds so a disconnected client does not leave an empty staging directory during capture. Validate on macOS/HVF and Linux/KVM with 118 live checks per host, independent qemu-img validation, crash recovery, and dirty-data hashes.
Require clock-aware generation activation for full checkpoints and wait for the kernel acknowledgement before thawing restored workloads. Skip the redundant initial agentd clock exchange after that barrier. Update the bundled kernel pointer and snapshot guidance, and add live clock/timer smoke fixtures. Older unreleased full artifacts must be recaptured; ordinary boot and file-only snapshots remain supported.
Define base-inclusive compaction counts that exclude the writable head, and immutable export suffixes with exact physical-baseline matching. Reject invalid counts instead of silently clamping and preserve an explicit dependency prefix for partial exports. This is shared planning groundwork only; disk mutation and dependent archive execution are not yet enabled. All 236 image tests pass, with four existing tests ignored.
Add user-controlled sealed-prefix compaction and dependent disk-layer archives across the CLI and Rust, Python, TypeScript and Go SDKs. Keep old snapshots immutable and resolve archive bases explicitly. Serialize disk replacement through the existing runtime control path. Retain stopped-disk ownership across cancelled awaits and preserve restart bindings after consolidation into a standalone qcow2 base. Include live managed/flat qualification, SDK smoke scripts, measured pause/export/restore samples, and an explicit remaining-coverage report. No newer main commits or dependency version changes are included.
|
Implemented explicit sealed-prefix compaction and incremental layer exports in ed4167f, stacked directly on the existing PR tip (no newer main). CLI and Rust/Python/TypeScript/Go interfaces include compaction, since/last-layer selection, explicit-base load, and direct archive restore. Live qualification: managed/flat running and stopped workflows pass on macOS/HVF, Linux/KVM, and Windows/WHP. macOS and Linux main matrices each passed 108 checks. Deep-chain 65 → 34 → 2 tests pass on both layouts on macOS/Linux and managed Windows, including restart and old-snapshot preservation. Windows flat-depth completion is unconfirmed after SSH became unreachable. Native language SDK workflows and cancellation/start races were live-tested on macOS. Short-chain VM pause samples (managed / flat): macOS 16.1 / 15.1 ms, Linux 2.20 / 2.28 ms, Windows 5.25 / 4.58 ms. These are measured samples, not latency guarantees. Full coverage, performance, reproduction scripts, and remaining qualification. Exhaustive power-loss/ENOSPC injection and the full SDK × OS matrix are not claimed complete. |
TL;DR
Capture a running sandbox as a full snapshot and restore its disk, memory, execution, device, and runtime filesystem state. The same artifact can also be restored disk-only for an ordinary cold boot.
Description
Test Plan
cargo fmt --all -- --checkpassesCloses #250