Skip to content

Show history import size, speed and activity - #1182

Merged
frahlg merged 3 commits into
masterfrom
fix/history-migration-progress
Sep 8, 2026
Merged

frahlg merged 3 commits into
masterfrom
fix/history-migration-progress

Conversation

@frahlg

@frahlg frahlg commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

History import can look stalled while Core checks source files, waits for live writes or checkpoints. After a restart, an already completed SQLite import also counts the entire old sample table again before reporting progress.

Show compressed Parquet source MB processed and remaining, speed, elapsed time, estimated time left, and the current activity in the history panel. Count previously imported bytes toward progress but exclude them from the current run's speed. Mark partial-file bytes as estimates based on verified rows. Omit unknown totals and withhold ETA until enough new work has completed; hide stale or inactive estimates in the UI.

Read the saved SQLite completion receipt before touching the old sample table. Legacy receipts without a saved row count keep that count unknown. When source bytes are available, use them for overall progress instead of presenting the incomplete legacy row count as a total. Keep the current file's verified rows and file count visible.

Validation:

  • Integrated make verify passed (vet, tests and build).
  • Focused race tests passed for progress estimates, completed SQLite imports without a SQLite read, compressed source byte counts, beta.1 resume and abrupt process exit.
  • The UI task checked the rendered progress and checking states in a browser, including the accessibility tree and screenshot; its nine focused UI tests passed. Human visual review remains part of the beta check.

This PR changes import status reporting and the completed-import shortcut. The separately observed slow year query during background import still needs follow-up; these checks do not establish physical-device migration completion or a query speedup.


Note

Medium Risk
Changes history migration status computation and Parquet import preamble (checkpoint vs prior rotation), plus restart behavior for completed SQLite imports; import correctness is covered by new tests but timing/rotation differences could affect long-running migrations.

Overview
History import now exposes richer status while Core migrates legacy SQLite and Parquet: compressed archive MB processed/remaining, throughput, elapsed time, and ETA when there is enough recent import work. The API adds activity (checking, importing, waiting_for_live, checkpointing) and omits byte totals or ETA when sizes are unknown, prior bytes would skew the rate, or the importer is not actively reading archives.

On the backend, Parquet progress is driven by on-disk compressed file sizes with row-fraction estimates for the file in flight; checkpoints temporarily surface as checkpointing. After SQLite import is already marked complete, restarts load rows_done from DuckDB progress instead of re-scanning ts_samples (legacy installs without a saved cursor keep row totals unknown). The history import UI prefers byte-based progress for the archive phase, shows the current step and metrics line, and suppresses speed/ETA when disconnected, failed, stale, or blocked on live writes.

Reviewed by Cursor Bugbot for commit 891dfb2. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-08T11:47:10.759690Z 891dfb2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cursor

cursor Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_d5d0e60a-e018-42af-995a-1e88bdc9d30d)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 891dfb2318

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread web/history-migration.js
waiting_for_live: "Waiting for new measurements to be saved",
checkpointing: "Saving database progress",
};
const step = !connected ? "" : steps[migration.activity] || "";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Hide active-step text after import failure

When an import fails, runHistoryMigration changes only State and LastError, leaving Activity at values such as importing, checking, or checkpointing. Because this line suppresses the step only when disconnected, a connected failed migration displays “History import paused” together with an ongoing-action message such as “Importing archive readings,” which falsely suggests that work is still progressing. Suppress the step when failed is true or clear the activity in the failure transition.

Useful? React with 👍 / 👎.

@frahlg
frahlg merged commit 419c857 into master Sep 8, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant