logreader: read segments in order, not discovery order - #38686
Conversation
auto_source accumulates valid_files across the filename and source loops, pruning needed_seg_idxs each pass, so segments resolved by a later pass land at the end of the dict. Returning .values() gave insertion order, so a route whose segments come from mixed sources was read out of order and logMonoTime moved backwards across segment boundaries. sort_by_time does not cover this; it sorts within one file. Return the files ordered by segment index. seg_idxs is always ascending, since a descending slice always evaluates to empty.
Process replay diff reportReplays driving segments through this PR and compares the behavior to master. ✅ 0 changed, 66 passed, 0 errors |
|
The two red checks are the fork-runner GL gap, not this change. Because this PR comes from a fork, the runner selection falls through to GitHub's hosted image (
I also ran this workflow against untouched master on my fork as a baseline, and it returns the identical Everything that does exercise the change passes: |
Description
auto_sourceaccumulatesvalid_files: dict[int, str]across two nested loops: filenames (rlog, then qlog) atlogreader.py:169and sources at:170. Each pass prunesneeded_seg_idxsdown to the segments still missing (:178), so a later pass only ever adds the gaps, and those entries land at the end of the dict. The return handed back.values(), which is insertion order, not segment order:Whenever a lower-numbered segment is resolved after a higher-numbered one, the returned list is out of order. Two ways that happens in normal use:
"N/M rlogs were not found, falling back to qlogs for those segments..."(:192), so it is an expected case, not an exotic one.:249) and each returns only the segments it has. If the first source is missing an early segment that a later source supplies, the order is wrong.LogReader.__iter__(:268) reads the list in the order given, so segments come out shuffled andlogMonoTimemoves backwards across segment boundaries.sort_by_timedoes not cover this: it sortsself._entswithin a single file (:120).The existing tests cannot observe it.
local_sourceandlocal_auto_sourcereturn the same path for every segment viadict.fromkeys(test_logreader.py:84,:89), so ordering is invisible, andlocal_auto_sourcereturns{}for rlogs, which makes the qlog pass fill every segment in one call and therefore in order.test_sort_by_time(:254) reads a single file.Fix: return the files ordered by segment index.
seg_idxsis always ascending, since a descending slice always evaluates to empty (10:0:-1and0:6:-2both give[]), so this is exactly the requested order and can never reorder a valid request.Verification
Three segments where segment 1 has no rlog, resolved through a fake source, so no network or device is needed:
End to end, same setup with real logs written by
save_logand read back throughLogReader(..., sort_by_time=True):Also reproduced with no qlog fallback at all: two rlog-only sources, the first holding segments 1 and 2 and the second holding segment 0, returned
[seg1, seg2, seg0]before and[seg0, seg1, seg2]after.openpilot/tools/lib/tests/test_logreader.py: 56 tests pass (22 skipped, network).ruff,ty, andcodespellclean.