Skip to content

A crashed run's hot journal makes workflow inspection throw raw, and ends list for every run #513

Description

@taras

What happens

A host killed mid-transaction leaves its run store with a hot rollback journal.
SQLite refuses to read past one on a connection that is not allowed to write the
rollback, and the workflow inspection surface meets that refusal unprepared:

  • withSnapshot (packages/workflow/src/deno/lifecycle.ts) opens readonly on
    purpose — recognition's never-writes proof is staked on that connection.
  • refusal() classifies through translateSqliteError
    (packages/workflow/src/deno/schema.ts), which recognizes only
    SQLITE_NOTADB and SQLITE_CORRUPT. The SQLITE_READONLY family falls
    through the default and is re-thrown raw.
  • listRuns deliberately ends the whole request on one unreadable candidate:
    "a list is a claim about every run this root holds."

Put together: after any host crash that leaves a hot journal, xmd workflow status <id> on that run — and xmd workflow list across every run — throw
an untyped attempt to write a readonly database until some write-capable
owner happens to resume that one run. Crash recovery on the read path has no
owner and no classification, and the state is not damage: the store is a
perfectly healthy database one rollback away from readable.

How it was found

Tier FE's FEW1 tripped over exactly this state through its own readonly test
reader (#511, red main #510). PR #512 made the test observation deterministic,
which is instrumentation, not a fix for this surface — review feedback on that
PR is what named the underlying gap.

Reproducing

Any writer killed while a transaction has already touched the database file
(spilled pages, or the kill landing inside commit) leaves the state. A
deterministic recipe: PRAGMA cache_size = 1, BEGIN IMMEDIATE, insert past
the cache so pages spill, SIGKILL. A readonly SELECT then refuses with
attempt to write a readonly database; a write-capable connection's first read
plays the journal back and sees only committed rows.

The decision to settle

The constraint: withSnapshot's never-writes promise conflicts with SQLite
requiring a write to get past a hot journal. The coherent options:

  • (a) Classify, don't recover — teach translateSqliteError the
    SQLITE_READONLY family and answer a typed refusal ("this run's store awaits
    recovery by its next owner"). Minimal and keeps the promise; list fails
    with a name instead of a raw throw, but one crashed run still ends the
    listing.
  • (b) Recover on a copy — inspection copies the database and journal to
    scratch, lets SQLite recover the copy, reads it, discards it. The real store
    is never written; a crashed run is inspectable and listable. Costs a copy per
    crashed candidate, and the answer describes a store the caller does not hold
    a lock on.
  • (c) Recover the real store from inspection — simplest operator
    experience, but it breaks the recognition-never-writes proof the docstring
    relies on.

This is an architecture decision (what inspection is authorized to do, and what
list claims when one run is mid-recovery); it should be settled before code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions