What happens
A host killed mid-transaction leaves its run store with a hot rollback journal.
SQLite refuses to read past one on a connection that is not allowed to write the
rollback, and the workflow inspection surface meets that refusal unprepared:
withSnapshot (packages/workflow/src/deno/lifecycle.ts) opens readonly on
purpose — recognition's never-writes proof is staked on that connection.
refusal() classifies through translateSqliteError
(packages/workflow/src/deno/schema.ts), which recognizes only
SQLITE_NOTADB and SQLITE_CORRUPT. The SQLITE_READONLY family falls
through the default and is re-thrown raw.
listRuns deliberately ends the whole request on one unreadable candidate:
"a list is a claim about every run this root holds."
Put together: after any host crash that leaves a hot journal, xmd workflow status <id> on that run — and xmd workflow list across every run — throw
an untyped attempt to write a readonly database until some write-capable
owner happens to resume that one run. Crash recovery on the read path has no
owner and no classification, and the state is not damage: the store is a
perfectly healthy database one rollback away from readable.
How it was found
Tier FE's FEW1 tripped over exactly this state through its own readonly test
reader (#511, red main #510). PR #512 made the test observation deterministic,
which is instrumentation, not a fix for this surface — review feedback on that
PR is what named the underlying gap.
Reproducing
Any writer killed while a transaction has already touched the database file
(spilled pages, or the kill landing inside commit) leaves the state. A
deterministic recipe: PRAGMA cache_size = 1, BEGIN IMMEDIATE, insert past
the cache so pages spill, SIGKILL. A readonly SELECT then refuses with
attempt to write a readonly database; a write-capable connection's first read
plays the journal back and sees only committed rows.
The decision to settle
The constraint: withSnapshot's never-writes promise conflicts with SQLite
requiring a write to get past a hot journal. The coherent options:
- (a) Classify, don't recover — teach
translateSqliteError the
SQLITE_READONLY family and answer a typed refusal ("this run's store awaits
recovery by its next owner"). Minimal and keeps the promise; list fails
with a name instead of a raw throw, but one crashed run still ends the
listing.
- (b) Recover on a copy — inspection copies the database and journal to
scratch, lets SQLite recover the copy, reads it, discards it. The real store
is never written; a crashed run is inspectable and listable. Costs a copy per
crashed candidate, and the answer describes a store the caller does not hold
a lock on.
- (c) Recover the real store from inspection — simplest operator
experience, but it breaks the recognition-never-writes proof the docstring
relies on.
This is an architecture decision (what inspection is authorized to do, and what
list claims when one run is mid-recovery); it should be settled before code.
What happens
A host killed mid-transaction leaves its run store with a hot rollback journal.
SQLite refuses to read past one on a connection that is not allowed to write the
rollback, and the workflow inspection surface meets that refusal unprepared:
withSnapshot(packages/workflow/src/deno/lifecycle.ts) opens readonly onpurpose — recognition's never-writes proof is staked on that connection.
refusal()classifies throughtranslateSqliteError(
packages/workflow/src/deno/schema.ts), which recognizes onlySQLITE_NOTADBandSQLITE_CORRUPT. TheSQLITE_READONLYfamily fallsthrough the
defaultand is re-thrown raw.listRunsdeliberately ends the whole request on one unreadable candidate:"a list is a claim about every run this root holds."
Put together: after any host crash that leaves a hot journal,
xmd workflow status <id>on that run — andxmd workflow listacross every run — throwan untyped
attempt to write a readonly databaseuntil some write-capableowner happens to resume that one run. Crash recovery on the read path has no
owner and no classification, and the state is not damage: the store is a
perfectly healthy database one rollback away from readable.
How it was found
Tier FE's FEW1 tripped over exactly this state through its own readonly test
reader (#511, red
main#510). PR #512 made the test observation deterministic,which is instrumentation, not a fix for this surface — review feedback on that
PR is what named the underlying gap.
Reproducing
Any writer killed while a transaction has already touched the database file
(spilled pages, or the kill landing inside commit) leaves the state. A
deterministic recipe:
PRAGMA cache_size = 1,BEGIN IMMEDIATE, insert pastthe cache so pages spill, SIGKILL. A readonly
SELECTthen refuses withattempt to write a readonly database; a write-capable connection's first readplays the journal back and sees only committed rows.
The decision to settle
The constraint:
withSnapshot's never-writes promise conflicts with SQLiterequiring a write to get past a hot journal. The coherent options:
translateSqliteErrortheSQLITE_READONLYfamily and answer a typed refusal ("this run's store awaitsrecovery by its next owner"). Minimal and keeps the promise;
listfailswith a name instead of a raw throw, but one crashed run still ends the
listing.
scratch, lets SQLite recover the copy, reads it, discards it. The real store
is never written; a crashed run is inspectable and listable. Costs a copy per
crashed candidate, and the answer describes a store the caller does not hold
a lock on.
experience, but it breaks the recognition-never-writes proof the docstring
relies on.
This is an architecture decision (what inspection is authorized to do, and what
listclaims when one run is mid-recovery); it should be settled before code.