Repository navigation
fix(updater): protect long first DuckDB migrations from image-only rollback #1164
Description
The first DuckDB migration in v3.2.0-beta.1 can exceed the updater’s fixed 30-minute Core readiness limit on a Raspberry Pi with a microSD card. The current updater would then restore the previous image without restoring its matching state. That is unsafe after a schema-3 migration has started.
Observed on a Raspberry Pi 4:
- New Core started at 08:43:14 UTC. SQLite migration and verification finished at 08:57:11.
- The site has about 21 million SQLite samples and 140 million older Parquet rows in 131 files.
- Large Parquet files import at roughly 0.9 million rows per minute. The import continues without errors; RSS is about 196 MiB and Core has no restarts.
/api/healthcorrectly reports startup, while/api/statusstays unavailable until the full API is ready.- We stopped only the updater before its 30-minute timeout. Core continues importing. No automatic rollback occurred in this observation.
The release has been returned to draft while we resolve this upgrade path. Its tag and images remain intact for the supervised physical test.
Required fix:
- Allow a valid long first migration to finish without declaring the Core API ready early.
- Do not automatically boot an older Core against potentially changed state after a failed update. Recovery must pair a full backup with its matching image.
- Cover the first upgrade from older updater versions: the old updater currently replaces itself only after the new Core becomes ready. Changing only the new updater’s timeout does not fix that first hop.
- Show clear migration or recovery status. The old loaded UI currently displays a red
startingerror with a generic reload suggestion.
This blocks broader rollout and stable promotion. See #1129 for the migration and its completed data checks. Longer full-scale soak and portable restore tests are outside this immediate fix.
Activity
frahlg commentedon Sep 8, 2026
The physical gate found a second blocking failure in v3.2.1-beta.1. Core starts, all three drivers come online, and native planning/control run while the import resumes. However, the 128 MB history instance exhausts memory after the first durable telemetry tick. Later ticks remain queued; the writer retry does not rotate the native instance because maintenance runs only after a successful commit. The importer waits for that queue to drain.
At 10:42 UTC: 56 accepted ticks, 1 committed, 55 pending, 0 rejected. The failed operation writes an EV energy-ledger observation. A concurrent 365-day series query also returned HTTP 500; day/month and raw 30-day reads passed. The 1,038 interior CSV rows match exactly; 14 differences occur only in the two boundary buckets and remain recorded separately.
The release has returned to draft and no Discord announcement was sent. The fixed updater remains running and no image rollback occurred. A focused writer-recovery fix and a full telemetry/ledger test are in progress. The current pending ticks are in RAM; the running version has no supported maintenance API to drain them before restart.
frahlg commentedon Sep 8, 2026
A normal restart during the v3.2.2 to v3.2.3 update exposed a second cause of writer OOM. Live writes stopped after one new commit. The queue reached 64 pending ticks and started rejecting new ticks; the importer waited at 87 of 131 archive files. More than 20 successful database maintenance runs did not recover the queued transaction.
The transaction spreads several returning energy-counter deltas across roughly a day of five-minute buckets. One INSERT per bucket retains too much transaction memory. A small fresh-database reproduction with five counter flows and 161 samples fails at the unchanged 256 MB limit. PR #1184 writes each interval with one INSERT SELECT and passes that reproduction, full-tick rollback, cursor checks and same-receipt retry without duplicate energy. It also makes the import banner show the write error and its effect on new history.
The previous claim that the twelve-flow fixture represented only retired identities was too strong: startup can briefly select the old MAC identity before the serial identity is registered. The long-gap case is relevant to the real startup path and to normal meter outages.
Both v3.2.2-beta.1 and v3.2.3-beta.1 are held as draft releases. Their tags and images remain intact. No Discord announcement or image rollback occurred. Pending ticks remain volatile; gaps while writes were blocked are recorded as a limitation. Required local verification passes for #1184; CI and physical recovery are still pending. This issue remains open.
frahlg commentedon Sep 18, 2026
Closing: the current updater no longer does what this issue describes.
componentHealthTimeoutingo/cmd/ftw-updater/main.go), with the comment that first-start imports on microSD can take hours.startingwith migration progress instead of a generic reload error (setBootHealthinweb/update-badge.js).What remains is the first hop from a pre-3.2 updater that still has the 30-minute limit; that is tracked in #1223. The writer OOM findings in the comments were fixed in the 3.2.x line and are covered by the storage issues (#1275–#1280).