A failed transaction-log append orphans its partial bytes, baking a mid-file framing break into the log

[kriszyp]

Summary

A transaction-log append that fails part-way through leaves the bytes it already wrote on disk, unaccounted for, and the next successful append lands after them. The result is a permanently torn frame in the middle of the file with valid entries on both sides — the one shape open-time recovery deliberately refuses to repair.

This is distinct from #572 (partial writev bookkeeping, fixed in 2.0.0/1.4.1) and from the torn tail that recoverTail() handles. Both of those are closed; this path is still live on main.

Mechanism

Four pieces compose:

1. The log fd is opened O_RDWR | O_CREAT | O_APPEND (transaction_log_file_posix.cpp:128), so every write lands at the current physical end of file regardless of TransactionLogFile::size.
2. writeBatchToFile returns -1 on a hard error and discards totalWritten (transaction_log_file_posix.cpp). Bytes that already reached the file are never reported to the caller.
3. writeEntriesV1 throws on -1 and does not advance this->size, and does not remove the bytes that landed (transaction_log_file.cpp:365-371).
4. The next successful append goes to physical EOF — i.e. after the orphaned partial bytes.

So a single failed append leaves:

• a partial frame permanently embedded mid-file, with well-formed entries after it once writing resumes; and
• size under-reporting the physical extent by the orphan length from that point on.

recoverTail() classifies this as RecoveryScan::Kind::MidFileCorruption and leaves it intact by design (transaction_log_file.cpp:178) — correctly, since truncating would discard committed entries after the break:

Leave the file intact: entries are still framed after the break, so truncating would discard committed/replicated transactions. […] Reads past this point will fail until the file is repaired.

It also only ever runs on the active (highest-sequence) file, on the stated assumption that "rotated files are immutable and already complete" (transaction_log_store.cpp:1078). This path violates that assumption: the process survives the failed append and keeps writing, so the break gets baked into a file that later rotates and is never scanned again.

Why this matters

A full disk or exhausted quota is routine, and this converts a transient one into permanent, silent damage. Downstream, every reader stops at the break, so entries written and acknowledged after it become unreachable — HarperFast/harper#2016 (a table rolled back 2.2 days on crash-recovery replay) and HarperFast/harper#2063 (a replication stream starved for 11 days) are both this shape. Those are being addressed by teaching the readers to resync past a break, which recovers the entries but does not stop the tear from being created.

The invariant being violated is that a log file ends on an entry boundary. Everything downstream — the framing walk, recoverTail, the JS reader — assumes it.

Suggested fix

On a failed append, truncate the file back to size before throwing. size is the append-owned authoritative extent, the bytes past it were never acknowledged (the commit throws), and writeEntriesV1 holds fileMutex across the whole append, so no concurrent appender can have added bytes in the window. That restores the invariant instead of leaving a break for readers to cope with.

Windows needs the zero-fill equivalent rather than a truncate, for the same reason eraseTail does in #723: files are pre-extended and a zero timestamp is the end-of-entries marker.

Worth deciding alongside this: whether writeBatchToFile should report how many bytes it managed to write rather than collapsing to -1, so the caller can distinguish "nothing landed" from "a partial batch landed" without re-stating the file.

Related

• Partial writev in writeBatchToFile causes audit-log framing corruption #572 — partial writev bookkeeping (closed; a different producer of the same on-disk shape)
• No in-process recovery from a latched background error; only a restart clears it #730 — no in-process recovery from a latched background error, from the same quota-exhaustion incident
• Mid-log corrupt transaction-log frame silently truncates replay and replication — acknowledged writes lost harper#2016, Corrupt-log containment in RocksTransactionLogStore.getRange is per-drain, so one bad entry head-of-line-blocks a replication stream forever harper#2063 — the reader-side consequences

🤖 Generated with Claude Code

[kriszyp]

Fix in #751 — fix(txnlog): erase a failed append's partial bytes instead of baking a framing break into the log (draft, stacked on #723 — fix(txnlog): recover the committed watermark from the log tail on load, which supplies eraseTail()).

Also fixed there, found while reviewing this one: open()'s header initialization had the same unchecked-write shape — a header that only partly lands still set size to the full header length, and left a file that fails the "too small" check on every future open.

One thing this issue's "suggested fix" did not anticipate: the erase can itself fail (ENOSPC followed by EIO). Warning and continuing would recreate the break, so the file is now retired and the store rotates past it, keeping the orphan a repairable torn tail.