Skip to content

Isolate prefetched time-series windows before applying transforms - #40

Open
sylvesterkaczmarek wants to merge 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix/prefetch-window-isolation-20260928
Open

sylvesterkaczmarek wants to merge 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix/prefetch-window-isolation-20260928

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Fixes #39.

Change

Copy the input and output windows selected from a prefetched time series before applying transformations or returning the record. The existing slices share memory with the cache, so in-place preprocessing or edits to a returned sample corrupt later reads. In non-sequential mode the overlapping input/output windows can also change each other within one record.

The two expression changes isolate only the requested windows, without copying the entire cached dataset. The non-prefetched path, values before transformation, shapes, dtypes, output offsets, bounds checks and transform order are unchanged. Prefetched reads now allocate independent sample arrays, matching the ownership behavior of the TensorStore-read path.

Verification

All 22 new cases and all 29 existing data-source tests pass locally on macOS/Python 3.11. Tests use real in-memory TensorStore arrays, Grain transformations and a real Grain DataLoader over two epochs. They cover sequential and overlapping windows, offsets, float32/int32 data, repeated reads, caller-side mutations, in-place transformations, preserved cache values and existing invalid-index errors. No network-backed data source is used.

Exact-commit hosted validation tests 64f48e1d1d5e7ebf99249ca010f0a57d0ed5bb9e on Ubuntu/Python 3.11 and 3.12 with NumPy 1.26.4, Grain 0.2.3, JAX 0.4.38 and TensorStore 0.1.85. Both jobs pass all 51 selected tests, reproduce ten failures and twelve passing controls with original production code, verify the 29 pre-existing tests on the original source, restore the submitted bytes and rerun the focused tests. Source-distribution and wheel builds pass, and all 22 new cases pass against the installed wheel from outside the checkout with its import path verified.

New-test formatting, correctness lint and focused dependency checks pass. JUnit reports, logs, coverage and built packages are retained as workflow artifacts. The validation workflow is on a separate fork-only branch.

Scope

Only the data-source module and one new regression module change. The tests install focused data-loading dependencies; the complete training environment, scientific benchmarks, GPU execution and external datasets were not exercised. No production dependency, upstream workflow or existing test expectation changes are included.

@sylvesterkaczmarek

sylvesterkaczmarek commented Oct 7, 2026 •

Copy link
Copy Markdown
Author

@jan-matthis The current heads of #40, #43 and #45 are mergeable, all current checks are green, and there are no unresolved review threads. #40 fixes mutation of prefetched time-series windows, #43 preserves the frame axis for unbatched-video SSIM, and #45 decodes one- and two-class forecasts to finite bin centers. I am grouping the request here to avoid separate pings. Could you review them, or redirect any one if needed?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

In-place transformations corrupt prefetched time-series data

1 participant