cli: support converting PX4 ULog (.ulg) files to MCAP - #1751
Draft
amacneil wants to merge 2 commits into
Draft
Conversation
Add a hand-rolled ULog parser to `mcap convert`, dispatched on the .ulg/.ulog extension alongside ROS 1 bag and ROS 2 db3 inputs. ULog data is self-describing, so conversion uses the embedded message formats and needs no external definitions. Mapping (px4 profile): - uORB data topics -> protobuf messages, schema px4.<message_name>, topic <message_name>/<multi_id>, with multi_id in channel metadata. Schemas are built dynamically from ULog format definitions (nested types, fixed arrays, char[] strings, padding handling). - Logged strings (L/C) -> a single log_message topic using px4.log_message (timestamp, severity, text, tag); the kernel severity is normalized from the ASCII digit to 0-7. - Parameters (P/Q) -> px4.parameter messages on a parameters topic (initial snapshot plus runtime changes). - Info (I/M) -> an MCAP info metadata record. Timestamps are recorded relative to system boot. Co-authored-by: Adrian Macneil <adrian@foxglove.dev>
- Build a minimal per-schema FileDescriptorSet (the message plus its transitive nested deps) instead of embedding every format in every schema record. Cuts uncompressed schema bytes from O(types x channels) to O(types) and fixes a latent crash when a uORB format collides with the synthetic log_message/parameter names (each now lives in its own descriptor set). - Avoid deep-cloning the FormatDef on every data message by scoping the immutable borrow so decode finishes before the &mut self write. - Distinguish default parameters (Q) from actual values (P) via a 'default' field on px4.parameter instead of silently emitting duplicates. - Drop dead code in parse_param. - Add tests for info->metadata (incl. skipping binary blobs), incompat flag rejection, the tolerated DATA_APPENDED flag, and P-vs-Q defaults; document the fixture channel count. Co-authored-by: Adrian Macneil <adrian@foxglove.dev>
Contributor
There was a problem hiding this comment.
The five inline threads from the first pass are addressed and resolved. Two open questions from that review still stand:
- We now ship two ULog→MCAP converters with incompatible output — this CLI and
python/examples/ulog2mcap(px4.vsulog.namespace,log_messagevs/log_messagetopic, protobuf vs JSON params, raw kernel severity 0–7 vs mapped Foxglove Log levels). What's the user story? Is the Python example getting realigned or deprecated, or do we knowingly hand people two tools with divergent output? At minimum, a docs pointer naming the CLI as canonical would save confusion. - Timestamps are boot-relative, so the file opens at ~1970 in players.
python/examples/ulog2mcap/convert.pygrew a--start-dateoffset precisely because near-epoch timestamps look broken. Is boot-relative-only acceptable for v1, or do we want the offset flag before this leaves draft? It's the first thing a user notices opening the result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds PX4 ULog (
.ulg) support tomcap convert, dispatched on the file extension alongside ROS 1 bag and ROS 2 db3 inputs (addresses #1675). ULog is self-describing, so the converter uses the file's embedded message formats and needs no external definitions or sourced workspace.The decode is hand-rolled (no new dependency), in the same spirit as the existing
ros1_bagconverter. Output uses a newpx4profile.Mapping
D, described byFformats)px4.<message_name>; topic<message_name>/<multi_id>;multi_idin channel metadataL/C)log_messagetopic, schemapx4.log_message(timestamp,severity,text,tag)P/Q)px4.parametermessages on aparameterstopic (initial snapshot + runtime changes); adefaultfield marksQdefaults vsPactual valuesI/M)infometadata record (string/scalar fields; binary blobs skipped)Each schema carries its own minimal
FileDescriptorSet(the message plus its transitive nested dependencies), so schema bytes stay O(types) rather than O(types × channels). Schemas are built dynamically from the ULog format definitions, handling nested types, fixed-size arrays,char[]strings, and padding fields. proto3 lacks 8/16-bit integers so those widen to 32-bit;char[]becomes astring.Naming decisions
px4.namespace (notulog.): these are PX4/uORB message types that ULog merely records — analogous to not namespacing ROS bag messages by the container.log_messageuORB topic (timestamp/severity/text), andseverityis normalized from the on-disk ASCII digit ('0'–'7') to the integer kernel level0–7. No lossy display mapping is baked in; a Foxglove-sidepx4.log_message → foxglove.Logconverter is intended as a separate workstream./0,/1, …).multi_idis the uORB instance index (e.g. IMU #N on one vehicle), also surfaced in channel metadata.Timestamps are recorded relative to system boot (ULog stores microseconds since startup), preserving the raw values; absolute-time signals (
time_ref_utc, GPStime_utc_usec) are preserved in the output for any downstream consumer that wants to derive epoch.Testing
P-vs-Qdefaults, info→metadata (including skipping binary blobs), and incompatible-flag rejection vs the toleratedDATA_APPENDEDflag.python/examples/ulog2mcap/fixtures/test_ulog.ulg.cargo test -p mcap-cli,cargo clippy -p mcap-cli --all-targets -- --no-deps -D warnings,cargo fmt, andcspellall pass.