Introduce HDF file cache - #83
Open
pmrv wants to merge 13 commits into
Open
Conversation
Do the check and retrieve in the same lock context. Monkey patching as well.
Also rename context manager
Wrap handles in a nullcontext so that calling code can use >>> with open_hdf(...) as handle: ... ... as before without accidentally closing the file handle
If passed a str, the previous code opened the file, but didn't close or return it, therefor leaking it.
Instead I'll add a code path in open_hdf that checks if the file is closed and reopens under a warning as necessary.
Treat compatible modes correctly and do not mix a/r+.
Using this method the clean up is called after tearDown and that causes an error on windows.
for more information, see https://pre-commit.ci
Pull Request Test Coverage Report for Build 11256387758Details
💛 - Coveralls |
jan-janssen
reviewed
Oct 9, 2024
Comment on lines
+324
to
+326
| store = h5py.File(name=filename, mode=mode, libver="latest", swmr=False) | ||
| if not store.swmr_mode: | ||
| store.swmr_mode = True |
Contributor
There was a problem hiding this comment.
Is this change necessary? The previous code was based on the example from the h5py documentation:
https://docs.h5py.org/en/stable/swmr.html#using-the-swmr-feature-from-h5py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With a new context manager
CachedHDF, users can request a file handle fromh5io_browserthat stays open for the duration of manager. The context manager can be nested. To implement this I added a new functionopen_hdf5and replaced all calls to_open_hdf5. By default this function does nothing but check whether a requested file is already in the cache. If instructed socache=Trueit uses the context manager to create a new cache entry and returns that.I augmented to test suite to run the tests where it makes sense under the caching context as well.