Skip to content

Preserve complete Arrow episodes and their timestep order - #352

Open
betacatsling wants to merge 1 commit into
Farama-Foundation:mainfrom
betacatsling:fix/arrow-episode-boundaries
Open

Preserve complete Arrow episodes and their timestep order#352
betacatsling wants to merge 1 commit into
Farama-Foundation:mainfrom
betacatsling:fix/arrow-episode-boundaries

Conversation

@betacatsling

Copy link
Copy Markdown

ArrowStorage consumes one-shot episode indices while building datasets, then tries to iterate them again. It also treats scanner batches as whole episodes, truncating trajectories longer than a batch. Reading complete per-episode tables fixes both issues; serializing writes in order prevents threaded writes from reordering timesteps.

Validation: 4 failed on the unchanged base; 123 passed after the fix, covering Arrow/Parquet long trajectories, one-shot indices, serialization and storage tests. Black/isort and diff checks passed. CPU/local storage only; no GPU, MuJoCo or remote dataset claims. The relevant API documentation is updated.

AI assistance was used for implementation and local verification.

Fixes #351.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Arrow episode reads lose generator requests and truncate long trajectories

1 participant