Skip to content

Issue 183 riscv64 checkpoint restore #192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sachin2605 wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Issue 183 riscv64 checkpoint restore #192

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cd5e7f3
checkpoint: port restore engine to riscv64
sachin2605 Aug 7, 2026
df8d89d
checkpoint: add riscv64 test equivalents for rearm, fpstate, and rest…
sachin2605 Aug 7, 2026
19a67c2
fix(checkpoint): address all PR #192 riscv64 review blockers
sachin2605 Aug 8, 2026
acf3a0d
fix(build): restore accidentally dropped cargo:rustc-env line
sachin2605 Aug 8, 2026
2abaef6
fix(checkpoint): cover riscv64 in rearm fallback cfg gate
sachin2605 Aug 8, 2026
ea67952
fix(checkpoint): add STUB_BASE fallback for non-x86_64/riscv64 arches
sachin2605 Aug 8, 2026
979a640
fix: skip stub-window keep interval when STUB_BASE=0 (aarch64)
sachin2605 Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions crates/sandlock-core/build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,32 @@ fn main() {
// text and stack have to sit outside the address range programs occupy. The
// default -no-pie base (0x400000) is exactly where a static ET_EXEC workload
// loads, so the checkpoint's own text would be mapped over the running stub.
//
// Cross-compilation: when TARGET is riscv64gc-unknown-linux-gnu (or any
// riscv64* variant), look for a riscv64 cross-compiler. On the host it
// uses plain `cc` as before.
let stub_src = manifest_dir.join("src/checkpoint/restore-stub.c");
let out_dir = PathBuf::from(std::env::var("OUT_DIR").unwrap());
let stub_bin = out_dir.join("restore-stub");
let target = std::env::var("TARGET").unwrap_or_default();
let (ccs, warn) = if target.starts_with("riscv64")
|| target.starts_with("riscv64gc")
{
(
&["riscv64-linux-gnu-gcc", "riscv64-unknown-linux-gnu-gcc"][..],

@congwang-mk congwang-mk Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: a native riscv64 build loses the compiler entirely.

TARGET is riscv64gc-unknown-linux-gnu for a plain cargo build on a riscv64 host too, so the candidate list becomes just these two cross-prefixed names — neither of which exists on a riscv64 machine, where the compiler is /usr/bin/cc / /usr/bin/gcc (checked on our riscv64 test host). The stub is then never built, stub_links_at_the_reserved_base, the new synthetic test, and test_restore_glibc_vdso_program_resumes all silently return early, and restore_interactive fails at runtime with "restore-stub was not built" — on exactly the platform this PR is for.

Keep cc in the list, gated on HOST so a genuine cross build doesn't silently produce a host binary:

let host = std::env::var("HOST").unwrap_or_default();
// ... riscv64 branch:
if host.starts_with("riscv64") {
    &["cc", "riscv64-linux-gnu-gcc", "riscv64-unknown-linux-gnu-gcc"][..]
} else {
    &["riscv64-linux-gnu-gcc", "riscv64-unknown-linux-gnu-gcc"][..]
}

Minor, same hunk: || target.starts_with("riscv64gc") on line 39 is subsumed by target.starts_with("riscv64").

"cannot compile restore-stub for riscv64: its restore tests will be \
skipped. Install a riscv64 cross-compiler (e.g. riscv64-linux-gnu-gcc).",
)
} else {
(
&["cc"][..],
"cannot compile restore-stub: its restore tests will be skipped.",
)
};
build_static(
&stub_src,
&stub_bin,
&["cc"],
ccs,
&[
"-static",
"-nostdlib",
Expand All @@ -47,7 +66,7 @@ fn main() {
"-fno-tree-loop-distribute-patterns",
"-Wl,-Ttext-segment=0x30000000000",

@congwang-mk congwang-mk Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: 3 TiB is above the Sv39 user-address ceiling, so the stub cannot be exec'd on real riscv64 hardware.

restore_blob::STUB_BASE is 0x300_0000_0000 and this link address matches it, but Sv39 (what JH7110, SpacemiT K1, and most riscv64 Linux boards run) gives TASK_SIZE = 256 GiB. The PR's own new comment in resume.rs says addresses must stay below 0x40_0000_0000; this is 12x over that.

Verified on a riscv64 host (mmu: sv39, [stack] at 0x3ff1469000):

mmap STUB_BASE 0x300_0000_0000  -> ENOMEM
mmap test CODE 0x200_0000_0000  -> ENOMEM
mmap            0x2000_0000     -> ok

$ ./restore-stub          # ET_EXEC with p_vaddr = 0x300_0000_0000
Segmentation fault        # elf_map() fails past the point of no return

Relinking the same source at -Wl,-Ttext-segment=0x3000000000 (192 GiB, under the ceiling) makes it exec and run correctly — it reaches the blob parser and exits 3 on a short blob, as designed.

So riscv64 needs its own stub base inside the Sv39 window, chosen here and in restore_blob.rs:45 together (the stub_links_at_the_reserved_base test exists precisely to catch them drifting apart, and it will once the stub actually builds).

Aside, since it's the obvious suspicion with a 3 TiB text segment: -mcmodel is not a problem. GCC on riscv64 Linux defaults to medany, and la sp, stub_stack assembles to auipc/ld through the GOT — the link succeeds at either base.

],
"cannot compile restore-stub: its restore tests will be skipped.",
warn,
);
// Emit the path every run (rustc-env is not cached across build-script runs),
// whether or not the binary was just (re)built.
Expand Down
157 changes: 145 additions & 12 deletions crates/sandlock-core/src/checkpoint/restore-stub.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* restore-stub: freestanding self-restore stub (x86_64).
* restore-stub: freestanding self-restore stub (x86_64, riscv64).
*
* This is a core component of the checkpoint restore engine, not a test
* fixture: the supervisor execs this stub into a fresh, fully-sandboxed process
Expand All @@ -21,7 +21,8 @@
* 6. mprotect the anonymous regions down to their checkpointed protections;
* 7. unmap the leftovers of its own startup that the image did not overwrite;
* 8. reopen the fd table at its saved numbers and offsets;
* 9. restore the thread pointer, which the signal frame cannot carry;
* 9. (x86_64) restore fs_base/gs_base via arch_prctl; (riscv64) tp is carried
* in the signal frame gregs, so nothing to do;
* 10. rt_sigreturn into the checkpoint's register context.
*
* Two address-space hazards drive the layout, and both are why this file avoids
Expand All @@ -48,8 +49,8 @@
*
* Exit codes (all _exit): 2 blob read, 3 bad magic/version/size, 4 map region,
* 5 open region file, 6 ready write, 7 go read, 8 mprotect, 9 vdso mremap,
* 10 fd reopen, 12 sweep entry overlapping the stub's own image, 13 arch_prctl.
* rt_sigreturn does not return; if it does, exit 11.
* 10 fd reopen, 12 sweep entry overlapping the stub's own image, 13 arch_prctl
* (x86_64 only). rt_sigreturn does not return; if it does, exit 11.
*/
#define CTRL_FD 3
#define READY_FD 4
Expand All @@ -61,6 +62,21 @@
#define STUB_BASE 0x30000000000UL
#define STUB_SPAN 0x400000UL

#ifdef __riscv

@congwang-mk congwang-mk Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__riscv is also defined on RV32, where __riscv_xlen == 32. The u64-based struct sigctx/struct uctx mirrors, the SYS_* numbers, and the 32x8-byte sc_regs copy are all RV64-only, and the #else #error added below never fires there — so an RV32 build would compile silently wrong rather than being rejected.

Suggest #if defined(__riscv) && __riscv_xlen == 64.

#define SYS_read 63
#define SYS_write 64
#define SYS_close 57
#define SYS_lseek 62
#define SYS_mmap 222
#define SYS_mprotect 226
#define SYS_munmap 215
#define SYS_mremap 216
#define SYS_dup3 24
#define SYS_exit 93
#define SYS_openat 56
#define SYS_rt_sigreturn 139
/* No SYS_arch_prctl on riscv64 — tp is in the signal frame gregs. */
#elif __x86_64__
#define SYS_read 0
#define SYS_write 1
#define SYS_close 3
Expand All @@ -74,9 +90,15 @@
#define SYS_openat 257
#define SYS_rt_sigreturn 15
#define SYS_arch_prctl 158
#else
#error "unsupported architecture"
#endif

/* x86_64 only: thread-pointer restore constants. */
#ifdef __x86_64__
#define ARCH_SET_GS 0x1001
#define ARCH_SET_FS 0x1002
#endif

#define PROT_READ 0x1
#define PROT_WRITE 0x2
Expand All @@ -93,8 +115,13 @@
* bytes per mapping and the string table one copy of each distinct mapped
* path. Rust fails the restore rather than truncating if a blob exceeds it. */
#define CTRL_MAX (1 << 20)
/* Upper bound on a signal-frame FP image: AMX-sized xstate plus magic2. */
/* Upper bound on a signal-frame FP image. x86_64: AMX-sized xstate plus
* magic2. riscv64: 544 bytes for Q-extension __riscv_fp_state. */
#if defined(__x86_64__)
#define FP_MAX 16384
#elif defined(__riscv)
#define FP_MAX 544
#endif
#define STACK_SIZE 65536
/* Leftover mappings the supervisor may ask the stub to unmap. A freshly
* execve'd stub has only its own image and its startup stack, so this is far
Expand All @@ -110,6 +137,7 @@ typedef unsigned long u64;
typedef unsigned int u32;
typedef long i64;

#ifdef __x86_64__
static i64 sc6(long n, u64 a, u64 b, u64 c, u64 d, u64 e, u64 f) {
i64 r;
register u64 r10 __asm__("r10") = d;
Expand All @@ -120,6 +148,21 @@ static i64 sc6(long n, u64 a, u64 b, u64 c, u64 d, u64 e, u64 f) {
: "rcx", "r11", "memory");
return r;
}
#elif defined(__riscv)
static i64 sc6(long n, u64 a, u64 b, u64 c, u64 d, u64 e, u64 f) {
register long nr __asm__("a7") = n;
register u64 a0 __asm__("a0") = a;
register u64 a1 __asm__("a1") = b;
register u64 a2 __asm__("a2") = c;
register u64 a3 __asm__("a3") = d;
register u64 a4 __asm__("a4") = e;
register u64 a5 __asm__("a5") = f;
__asm__ volatile("ecall" : "+r"(a0)
: "r"(a1), "r"(a2), "r"(a3), "r"(a4), "r"(a5), "r"(nr)
: "memory");
return (i64)a0;
}
#endif
#define SC1(n,a) sc6(n,(u64)(a),0,0,0,0,0)
#define SC2(n,a,b) sc6(n,(u64)(a),(u64)(b),0,0,0,0)
#define SC3(n,a,b,c) sc6(n,(u64)(a),(u64)(b),(u64)(c),0,0,0)
Expand All @@ -144,8 +187,8 @@ void *memcpy(void *d, const void *s, unsigned long n) {
return d;
}

/* Blob layout mirror (little-endian; we run on x86_64 LE so struct reads work).
* Must match checkpoint/restore_blob.rs byte for byte. */
/* Blob layout mirror (little-endian; both x86_64 and riscv64 are LE, so native
* struct reads work). Must match checkpoint/restore_blob.rs byte for byte. */
struct blob_header {
u32 magic, version, n_regions, n_fds;
u64 regs_off; u32 regs_len, fpstate_len;
Expand All @@ -165,11 +208,15 @@ struct blob_vdso { i64 delta; u64 len; u64 target; };
#define SRC_ANON 0
#define SRC_FILE 1

/* ---- x86_64 rt_sigreturn frame -------------------------------------------
* rt_sigreturn reads the ucontext at rsp (kernel does frame = rsp - 8; uc is at
* frame+8 = rsp). We build a ucontext, set rsp to &uc, and syscall rt_sigreturn.
* mcontext gregs order (x86_64): see REG_* below.
*/
/* ---- Architecture-specific signal frame layouts --------------------------
* The stub builds the frame on its private stack and rt_sigreturns into it.
* Registers in the blob are in ptrace order (capture::ptrace_getregs). */

#ifdef __x86_64__

/* x86_64: rt_sigreturn reads the ucontext at rsp (kernel does frame = rsp - 8;
* uc is at frame+8 = rsp). We build a ucontext, set rsp to &uc, and syscall
* rt_sigreturn. mcontext gregs order: see REG_* below. */
enum { R8=0,R9,R10,R11,R12,R13,R14,R15,RDI,RSI,RBP,RBX,RDX,RAX,RCX,RSP,RIP,
EFL,CSGSFS,ERR,TRAPNO,OLDMASK,CR2 }; /* 23 gregs */
struct sigctx { u64 gregs[23]; u64 fpstate; u64 reserved[8]; };
Expand All @@ -193,11 +240,47 @@ enum { UR_R15=0,UR_R14,UR_R13,UR_R12,UR_RBP,UR_RBX,UR_R11,UR_R10,UR_R9,UR_R8,
UR_RAX,UR_RCX,UR_RDX,UR_RSI,UR_RDI,UR_ORIG_RAX,UR_RIP,UR_CS,UR_EFLAGS,
UR_RSP,UR_SS,UR_FS_BASE,UR_GS_BASE,UR_DS,UR_ES,UR_FS,UR_GS };

#elif defined(__riscv)

/* riscv64: rt_sigreturn reads frame at sp = (struct rt_sigframe *)sp.
* siginfo (128 bytes) + ucontext. uc_mcontext is at uc+0xA8 (168).
* sc_regs[32] at sigcontext+0x00, sc_fpregs at sigcontext+0x100.
* The gp[] blob order is ptrace: pc=0, ra=1, sp=2, gp=3, tp=4, t0-t2=5-7,
* s0-s1=8-9, a0-a7=10-17, s2-s11=18-27, t3-t6=28-31 — a 1:1 mapping to
* sc_regs[32], so no remap is needed. tp (thread pointer) is carried by the
* signal frame's sc_regs[4]; nothing needs arch_prctl. */
struct sigctx {
u64 gregs[32]; /* sc_regs: 32 gregs, 256 bytes */
u8 fpregs[544]; /* sc_fpregs: union __riscv_fp_state (max Q ext) */

@congwang-mk congwang-mk Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: this branch does not compile — u8 is never declared.

The typedefs at lines 136-138 are u64, u32, i64 only; u8 was never needed before this PR. Building the stub natively on riscv64 (Debian gcc 14, mmu: sv39) with exactly the flags build.rs passes:

/tmp/stub.c:254:5: error: unknown type name 'u8'
  254 |     u8  fpregs[544];
/tmp/stub.c:265:5: error: unknown type name 'u8'
/tmp/stub.c:271:5: error: unknown type name 'u8'

Adding typedef unsigned char u8; next to the others makes it compile and link.

This is why CI is green while none of the riscv64 work has ever run: build_static turns a compiler failure into a cargo:warning, and every riscv64 test then early-returns on !stub.exists().

};

struct uctx {
u64 uc_flags; /* 0x00 */
u64 uc_link; /* 0x08 */
u64 ss_sp; /* 0x10 */
u32 ss_flags; /* 0x18 */
u32 _pad; /* 0x1C */
u64 ss_size; /* 0x20 */
u64 uc_sigmask; /* 0x28 */
u8 __unused[120]; /* 0x30 */
struct sigctx mc; /* 0xA8 (168) */

@congwang-mk congwang-mk Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: uc_mcontext is at offset 176 (0xB0), not 168 (0xA8).

struct sigcontext has 16-byte alignment on riscv64 — both __riscv_q_ext_state.f[64] and __riscv_extra_ext_header.__padding carry __attribute__((aligned(16))), so the union and therefore struct sigcontext are 16-aligned. 168 is not a multiple of 16, so the kernel pads to 176. Measured on a riscv64 host against <asm/ucontext.h>:

offsetof(kernel struct ucontext, uc_mcontext) = 176 (0xb0)
sizeof(struct sigcontext) = 784

struct sigctx here has alignment 8, so the compiler places mc at exactly 168 and the whole register file lands 8 bytes low. rt_sigreturn then reads pc from gregs[1] (ra), sp from gregs[2] (gp), and so on. In the new synthetic test ra is 0, so the restored process jumps to 0 and dies.

Fix: __attribute__((aligned(16))) on struct sigctx (or widen __unused to 128 bytes and keep the comment honest).

};

/* rt_sigframe: struct siginfo (zeroed, 128 bytes) + ucontext. */
struct rt_sf {
u8 info[128];
struct uctx uc;
};

#endif

/* These all live in .bss at STUB_BASE, out of reach of any MAP_FIXED region.
* stub_stack is global so the module-level asm below can reference it. */
static char ctrl_buf[CTRL_MAX] __attribute__((aligned(16)));
#ifdef __x86_64__
/* xrstor requires the signal frame's FP image to be 64-byte aligned. */
static char fp_buf[FP_MAX] __attribute__((aligned(64)));
#endif
/* Interleaved (start, len) pairs of the mappings to shed. */
static u64 sweep[MAX_SWEEP * 2];
char stub_stack[STACK_SIZE];
Expand Down Expand Up @@ -356,12 +439,18 @@ static void _start_c(u64 *sp) {
i64 fd = SC4(SYS_openat, AT_FDCWD, strings + f->path_off, f->flags, 0);
if (fd < 0) die(10);
if ((u32)fd != f->fd) {
#ifdef __riscv
/* riscv64 has no SYS_dup2 — use dup3 with flags=0. */
if (SC3(SYS_dup3, fd, f->fd, 0) != (i64)f->fd) die(10);
#else
if (SC2(SYS_dup2, fd, f->fd) != (i64)f->fd) die(10);
#endif
SC1(SYS_close, fd);
}
SC3(SYS_lseek, f->fd, f->offset, SEEK_SET);
}

#ifdef __x86_64__
/* 9. Restore the thread pointer. The x86_64 signal frame has 23 gregs and
* none of them is fs_base, so rt_sigreturn cannot carry it and the resumed
* program would inherit this stub's, which is zero because a -nostdlib
Expand All @@ -373,10 +462,12 @@ static void _start_c(u64 *sp) {
* ordinary user programs; set it only when the checkpoint recorded one. */
if (SC2(SYS_arch_prctl, ARCH_SET_FS, gp[UR_FS_BASE]) != 0) die(13);
if (gp[UR_GS_BASE] && SC2(SYS_arch_prctl, ARCH_SET_GS, gp[UR_GS_BASE]) != 0) die(13);
#endif

/* 10. Build the rt_sigframe on our private stack and rt_sigreturn into the
* checkpoint. The frame must be readable when the kernel consumes it; the
* stub stack is a plain .bss mapping at STUB_BASE, so it always is. */
#ifdef __x86_64__
struct uctx uc;
memset(&uc, 0, sizeof uc);
struct sigctx *m = &uc.mc;
Expand Down Expand Up @@ -414,13 +505,41 @@ static void _start_c(u64 *sp) {
:
: "r"(&uc), "r"(rax)
: "memory");

#elif defined(__riscv)
/* riscv64: build a struct rt_sigframe on the stack. gp[] order is 1:1 with
* sc_regs (both ptrace order), so copy the register file directly.
* The FP state is embedded inline in sc_fpregs (no pointer indirection,
* no magic framing — restore_blob.rs sends the raw __riscv_d_ext_state). */
struct rt_sf sf;
memset(&sf, 0, sizeof sf);
/* gp has regs_len / 8 entries; copy all of them into sc_regs[32]. */
u32 nregs = h->regs_len / 8;
if (nregs > 32) nregs = 32;
memcpy(sf.uc.mc.gregs, gp, nregs * sizeof(u64));
if (h->fpstate_len) {
if (h->fpstate_len > sizeof(sf.uc.mc.fpregs))

@congwang-mk congwang-mk Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here:

  1. This clamp is dead. Line 349 already does if (h->fpstate_len > FP_MAX) die(3);, and FP_MAX is 544 on riscv64 — the same bound. It also mutates the header in place inside ctrl_buf, which the rest of the stub treats as read-only after validation.

  2. If it were reachable, 544 is the wrong bound. The kernel's second sigcontext member is union { union __riscv_fp_state sc_fpregs; struct __riscv_extra_ext_header sc_extdesc; }, which is 528 bytes, with sc_extdesc.reserved at offset 516 and sc_extdesc.hdr at 520 (verified: sizeof(struct sigcontext) == 784 = 256 + 528). A 544-byte copy would overwrite reserved, and restore_sigcontext() returns -EINVAL on a non-zero reserved, so rt_sigreturn would fail instead of resuming.

Not reachable today (riscv NT_PRFPREG yields 264 bytes), but FP_MAX should be 516/528 rather than 544, and the comment above it calls 544 the Q-extension size, which it isn't.

h->fpstate_len = (u32)sizeof(sf.uc.mc.fpregs);
memcpy(sf.uc.mc.fpregs, ctrl_buf + h->fpstate_off, h->fpstate_len);
}

/* Set sp = &sf, then ecall rt_sigreturn. */
register u64 a7 __asm__("a7") = SYS_rt_sigreturn;
__asm__ volatile(
"mv sp, %0\n\t"
"ecall\n\t"
:
: "r"(&sf), "r"(a7)
: "memory");
#endif
die(11); /* rt_sigreturn must not return */
}

/* No libc: provide the ELF entry. Hand the kernel-provided stack pointer to
* _start_c as its argument (auxv lives there), then switch to the private .bss
* stack, because the checkpoint's [stack] region is mapped over the address the
* kernel picked for ours. */
#ifdef __x86_64__
__asm__(
".global _start\n"
"_start:\n"
Expand All @@ -432,3 +551,17 @@ __asm__(
" call _start_c\n"
" hlt\n"
);
#elif defined(__riscv)
/* riscv64: a0 = sp (first argument), switch to stub_stack, align, call. */
__asm__(
".global _start\n"
"_start:\n"
" mv a0, sp\n"
" la sp, stub_stack\n"
" li t0, " STR(STACK_SIZE) "\n"
" add sp, sp, t0\n"
" andi sp, sp, -16\n"
" call _start_c\n"
" unimp\n"
);
#endif
Loading
Loading