Repository navigation
Isolate prefetched time-series windows before applying transforms - #40
Open
sylvesterkaczmarek wants to merge 1 commit into
Open
sylvesterkaczmarek wants to merge 1 commit into
sylvesterkaczmarek wants to merge 1 commit into
Conversation
Author
|
@jan-matthis The current heads of #40, #43 and #45 are mergeable, all current checks are green, and there are no unresolved review threads. #40 fixes mutation of prefetched time-series windows, #43 preserves the frame axis for unbatched-video SSIM, and #45 decodes one- and two-class forecasts to finite bin centers. I am grouping the request here to avoid separate pings. Could you review them, or redirect any one if needed? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #39.
Change
Copy the input and output windows selected from a prefetched time series before applying transformations or returning the record. The existing slices share memory with the cache, so in-place preprocessing or edits to a returned sample corrupt later reads. In non-sequential mode the overlapping input/output windows can also change each other within one record.
The two expression changes isolate only the requested windows, without copying the entire cached dataset. The non-prefetched path, values before transformation, shapes, dtypes, output offsets, bounds checks and transform order are unchanged. Prefetched reads now allocate independent sample arrays, matching the ownership behavior of the TensorStore-read path.
Verification
All 22 new cases and all 29 existing data-source tests pass locally on macOS/Python 3.11. Tests use real in-memory TensorStore arrays, Grain transformations and a real Grain DataLoader over two epochs. They cover sequential and overlapping windows, offsets, float32/int32 data, repeated reads, caller-side mutations, in-place transformations, preserved cache values and existing invalid-index errors. No network-backed data source is used.
Exact-commit hosted validation tests
64f48e1d1d5e7ebf99249ca010f0a57d0ed5bb9eon Ubuntu/Python 3.11 and 3.12 with NumPy 1.26.4, Grain 0.2.3, JAX 0.4.38 and TensorStore 0.1.85. Both jobs pass all 51 selected tests, reproduce ten failures and twelve passing controls with original production code, verify the 29 pre-existing tests on the original source, restore the submitted bytes and rerun the focused tests. Source-distribution and wheel builds pass, and all 22 new cases pass against the installed wheel from outside the checkout with its import path verified.New-test formatting, correctness lint and focused dependency checks pass. JUnit reports, logs, coverage and built packages are retained as workflow artifacts. The validation workflow is on a separate fork-only branch.
Scope
Only the data-source module and one new regression module change. The tests install focused data-loading dependencies; the complete training environment, scientific benchmarks, GPU execution and external datasets were not exercised. No production dependency, upstream workflow or existing test expectation changes are included.