Skip to content

feat(oak): make oak frame tracker track 1 stream #874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shaosu-nvidia wants to merge 1 commit into from
Closed

feat(oak): make oak frame tracker track 1 stream #874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions docs/source/device/oak.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,10 @@ controllers, etc.):
:doc:`TeleopSession <../getting_started/teleop_session>`'s ``PluginConfig``,
passing ``--collection-prefix`` so the plugin pushes metadata via OpenXR
``SchemaPusher``.
2. **Host tracker** — create ``FrameMetadataTrackerOak`` with the **same**
collection prefix and stream list (see :doc:`trackers`). TeleopSession's
DeviceIO layer uses the live tracker implementation to read the pushed
tensors and write MCAP channels.
2. **Host trackers** — create one ``FrameMetadataTrackerOak`` per stream, each
with a ``collection_id`` matching the plugin's ``{collection_prefix}/{StreamName}``
(see :doc:`trackers`). TeleopSession's DeviceIO layer uses the live tracker
implementation to read the pushed tensors and write MCAP channels.
3. **MCAP config** — add the tracker to both ``TeleopSessionConfig.trackers``
and ``McapRecordingConfig.tracker_names``. See also
:doc:`../references/mcap_record_replay`.
Expand All @@ -120,22 +120,25 @@ controllers, etc.):

from pathlib import Path

from isaacteleop.deviceio import FrameMetadataTrackerOak, McapRecordingConfig, StreamType
from isaacteleop.deviceio import FrameMetadataTrackerOak, McapRecordingConfig
from isaacteleop.teleop_session_manager import PluginConfig, TeleopSession, TeleopSessionConfig

PLUGIN_ROOT = Path("build/src/plugins") # or your installed plugin search path
COLLECTION_PREFIX = "oak_camera"
STREAMS = [StreamType.Color, StreamType.MonoLeft]
STREAM_NAMES = ["Color", "MonoLeft"]

oak_tracker = FrameMetadataTrackerOak(COLLECTION_PREFIX, STREAMS)
oak_trackers = [
FrameMetadataTrackerOak(f"{COLLECTION_PREFIX}/{name}")
for name in STREAM_NAMES
]

config = TeleopSessionConfig(
app_name="OakTeleop",
pipeline=pipeline, # your retargeting pipeline
trackers=[oak_tracker],
trackers=oak_trackers,
mcap_config=McapRecordingConfig(
"recording.mcap",
[(oak_tracker, "oak_metadata")],
[(t, f"oak_metadata/{name}") for t, name in zip(oak_trackers, STREAM_NAMES)],
),
plugins=[
PluginConfig(
Expand Down
5 changes: 3 additions & 2 deletions docs/source/device/trackers.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,14 +220,15 @@ reads the PICO ``XR_BD_body_tracking`` extension directly.
FrameMetadataTrackerOak
~~~~~~~~~~~~~~~~~~~~~~~

Multi-channel tracker for per-frame metadata from OAK camera streams.
Single-stream tracker for per-frame metadata from an OAK camera stream.
Create one instance per stream (e.g. ``"oak_camera/Color"``, ``"oak_camera/MonoLeft"``).
Uses the :code-file:`SchemaTracker <src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp>`
utility internally.

- Schema: :code-file:`src/core/schema/fbs/oak.fbs`
- C++ header: ``#include <deviceio/frame_metadata_tracker_oak.hpp>``
- Python import: ``from isaacteleop.deviceio import FrameMetadataTrackerOak``
- Record channels: one per configured stream (e.g. ``Color``, ``MonoLeft``) | MCAP schema: ``core.FrameMetadataOakRecord``
- Record channels: ``frame_metadata`` | MCAP schema: ``core.FrameMetadataOakRecord``
Comment on lines +223 to +231

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add the required SPDX header.

docs/source/device/trackers.rst lacks the repository SPDX copyright and license header. Add the standard reStructuredText SPDX block at the start of the file.

Proposed fix 
+.. SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+.. SPDX-License-Identifier: Apache-2.0
+
 Device Trackers

As per coding guidelines, “Files covered by REUSE policy must include the repository’s standard SPDX copyright and license headers.”

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/source/device/trackers.rst` around lines 223 - 231, Add the
repository-standard reStructuredText SPDX copyright and license header at the
beginning of the documentation file, before the existing tracker content.
Preserve all current documentation text and formatting after the header.

Source: Coding guidelines

- Tests:

- :code-file:`src/core/schema_tests/cpp/test_oak.cpp`
Expand Down
45 changes: 26 additions & 19 deletions examples/oxr/python/test_oak_camera.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@
import isaacteleop.deviceio as deviceio
import isaacteleop.oxr as oxr

PLUGIN_ROOT_DIR = Path(__file__).resolve().parent.parent.parent.parent / "plugins"
PLUGIN_ROOT_DIR = (
Path(__file__).resolve().parent.parent.parent.parent / "install/plugins"
)

MODE_NO_METADATA = "no-metadata"
MODE_SCHEMA_PUSHER = "schema-pusher"
Expand Down Expand Up @@ -55,21 +57,18 @@ def _run_recording_loop(plugin, duration: float):
def _run_schema_pusher(
plugin,
duration: float,
tracker,
trackers: list,
stream_names: list[str],
required_extensions: list[str],
mcap_filename: str,
recording_config: deviceio.McapRecordingConfig,
):
"""Read metadata via OpenXR schema tracker and record to MCAP on the host side."""
with oxr.OpenXRSession("OakCameraTest", required_extensions) as oxr_session:
handles = oxr_session.get_handles()
print(" ✓ OpenXR session created")

recording_config = deviceio.McapRecordingConfig(
mcap_filename, [(tracker, "oak_metadata")]
)
with deviceio.DeviceIOSession.run(
[tracker], handles, recording_config
trackers, handles, recording_config
) as session:
print(" ✓ DeviceIO session initialized (recording active during update())")
print()
Expand All @@ -88,8 +87,8 @@ def _run_schema_pusher(
frame_count += 1

elapsed = time.time() - start_time
for idx, name in enumerate(stream_names):
tracked = tracker.get_stream_data(session, idx)
for tracker, name in zip(trackers, stream_names):
tracked = tracker.get_data(session)
if (
tracked.data is not None
and tracked.data.sequence_number != last_seq.get(name, -1)
Expand Down Expand Up @@ -152,23 +151,23 @@ def run_test(duration: float = 10.0, mode: str = MODE_NO_METADATA):

# 3. Prepare mode-specific state
stream_names = ["Color", "MonoLeft"]
stream_types = [deviceio.StreamType.Color, deviceio.StreamType.MonoLeft]
collection_prefix = "oak_camera"
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
mcap_filename = f"camera_metadata_{timestamp}.mcap"

tracker = None
trackers = []
required_extensions = []

if mode == MODE_SCHEMA_PUSHER:
print("[Step 3] Creating composite FrameMetadataTrackerOak...")
tracker = deviceio.FrameMetadataTrackerOak(collection_prefix, stream_types)
print("[Step 3] Creating FrameMetadataTrackerOak instances (one per stream)...")
trackers = [
deviceio.FrameMetadataTrackerOak(f"{collection_prefix}/{name}")
for name in stream_names
]
print(
f" Created tracker (prefix: {collection_prefix}, streams: {stream_names})"
)
required_extensions = deviceio.DeviceIOSession.get_required_extensions(
[tracker]
f" Created {len(trackers)} trackers: {[f'{collection_prefix}/{n}' for n in stream_names]}"
)
required_extensions = deviceio.DeviceIOSession.get_required_extensions(trackers)
print()
print("[Step 4] Getting required OpenXR extensions...")
print(f" Required extensions: {required_extensions}")
Expand Down Expand Up @@ -207,13 +206,21 @@ def run_test(duration: float = 10.0, mode: str = MODE_NO_METADATA):
print(" Camera plugin started")

if mode == MODE_SCHEMA_PUSHER:
recording_config = deviceio.McapRecordingConfig(
mcap_filename,
[
(t, f"oak_metadata/{name}")
for t, name in zip(trackers, stream_names)
],
)

_run_schema_pusher(
plugin,
duration,
tracker,
trackers,
stream_names,
required_extensions,
mcap_filename,
recording_config,
)
else:
_run_recording_loop(plugin, duration)
Expand Down
59 changes: 21 additions & 38 deletions examples/schemaio/frame_metadata_printer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
#include <deviceio_session/deviceio_session.hpp>
#include <deviceio_trackers/frame_metadata_tracker_oak.hpp>
#include <oxr/oxr_session.hpp>
#include <schema/oak_generated.h>

#include <chrono>
#include <iostream>
Expand Down Expand Up @@ -67,39 +68,40 @@ try

std::cout << "Frame Metadata Printer (prefix: " << collection_prefix << ")" << std::endl;

// Track all three stream types; streams without a pusher simply won't receive data.
std::vector<core::StreamType> streams = { core::StreamType_Color, core::StreamType_MonoLeft,
core::StreamType_MonoRight };
// One tracker per stream; streams without a pusher simply won't receive data.
const std::vector<std::string> collection_ids = {
collection_prefix + "/Color",
collection_prefix + "/MonoLeft",
collection_prefix + "/MonoRight",
};

std::cout << "[Step 1] Creating FrameMetadataTrackerOak..." << std::endl;
auto tracker = std::make_shared<core::FrameMetadataTrackerOak>(collection_prefix, streams, MAX_FLATBUFFER_SIZE);
std::cout << "[Step 1] Creating FrameMetadataTrackerOak instances..." << std::endl;
std::vector<std::shared_ptr<core::FrameMetadataTrackerOak>> trackers;
trackers.reserve(collection_ids.size());
for (const auto& cid : collection_ids)
trackers.push_back(std::make_shared<core::FrameMetadataTrackerOak>(cid, MAX_FLATBUFFER_SIZE));

std::cout << "[Step 2] Creating OpenXR session with required extensions..." << std::endl;
std::vector<std::shared_ptr<core::ITracker>> trackers = { tracker };
auto required_extensions = core::DeviceIOSession::get_required_extensions(trackers);
std::vector<std::shared_ptr<core::ITracker>> itracker_list(trackers.begin(), trackers.end());
auto required_extensions = core::DeviceIOSession::get_required_extensions(itracker_list);
auto oxr_session = std::make_shared<core::OpenXRSession>("FrameMetadataPrinter", required_extensions);
std::cout << " OpenXR session created" << std::endl;

std::cout << "[Step 3] Creating DeviceIOSession..." << std::endl;
auto session = core::DeviceIOSession::run(trackers, oxr_session->get_handles());
auto session = core::DeviceIOSession::run(itracker_list, oxr_session->get_handles());

std::cout << "[Step 4] Reading samples (press Ctrl+C to stop)..." << std::endl;

size_t received_count = 0;

// Per-stream last-seen sequence number. nullopt means the stream has never
// Per-tracker last-seen sequence number. nullopt means the stream has never
// been observed — the first sample is always printed regardless of its value.
// If data is already present at startup we seed with its sequence so we don't
// reprint it; absent data stays nullopt so sequence 0 is never skipped.
size_t stream_count = tracker->get_stream_count();
std::vector<std::optional<uint64_t>> last_sequences(stream_count);
for (size_t i = 0; i < stream_count; ++i)
std::vector<std::optional<uint64_t>> last_sequences(trackers.size());
for (size_t i = 0; i < trackers.size(); ++i)
{
const auto& tracked = tracker->get_stream_data(*session, i);
const auto& tracked = trackers[i]->get_data(*session);
if (tracked.data)
{
last_sequences[i] = tracked.data->sequence_number;
}
}

auto last_status_time = std::chrono::steady_clock::now();
Expand All @@ -109,28 +111,9 @@ try
{
session->update();

// Refresh stream count and extend per-stream tracking if streams were added.
stream_count = tracker->get_stream_count();
if (last_sequences.size() != stream_count)
{
size_t old_count = last_sequences.size();
last_sequences.resize(stream_count);
// Newly added streams start as nullopt; seed with current sequence if
// data is already present so we don't reprint an existing sample.
for (size_t i = old_count; i < stream_count; ++i)
{
const auto& tracked = tracker->get_stream_data(*session, i);
if (tracked.data)
{
last_sequences[i] = tracked.data->sequence_number;
}
}
}

// Print one line per stream that has a new sample.
for (size_t i = 0; i < stream_count; ++i)
for (size_t i = 0; i < trackers.size(); ++i)
{
const auto& tracked = tracker->get_stream_data(*session, i);
const auto& tracked = trackers[i]->get_data(*session);
if (!tracked.data ||
(last_sequences[i].has_value() && tracked.data->sequence_number == last_sequences[i].value()))
{
Expand Down
4 changes: 1 addition & 3 deletions src/core/deviceio_base/cpp/inc/deviceio_base/frame_metadata_tracker_oak_base.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@

#include "tracker.hpp"

#include <cstddef>

namespace core
{

Expand All @@ -16,7 +14,7 @@ struct FrameMetadataOakTrackedT;
class IFrameMetadataTrackerOakImpl : public ITrackerImpl
{
public:
virtual const FrameMetadataOakTrackedT& get_stream_data(size_t stream_index) const = 0;
virtual const FrameMetadataOakTrackedT& get_data() const = 0;
};

} // namespace core
27 changes: 4 additions & 23 deletions src/core/deviceio_trackers/cpp/frame_metadata_tracker_oak.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@

#include "inc/deviceio_trackers/frame_metadata_tracker_oak.hpp"

#include <stdexcept>
#include <string>

namespace core
Expand All @@ -13,32 +12,14 @@ namespace core
// FrameMetadataTrackerOak
// ============================================================================

FrameMetadataTrackerOak::FrameMetadataTrackerOak(const std::string& collection_prefix,
const std::vector<StreamType>& streams,
size_t max_flatbuffer_size)
: collection_prefix_(collection_prefix), streams_(streams), max_flatbuffer_size_(max_flatbuffer_size)
FrameMetadataTrackerOak::FrameMetadataTrackerOak(const std::string& collection_id, size_t max_flatbuffer_size)
: collection_id_(collection_id), max_flatbuffer_size_(max_flatbuffer_size)
{
if (streams.empty())
{
throw std::runtime_error("FrameMetadataTrackerOak: at least one stream is required");
}

for (auto type : streams)
{
const char* name = EnumNameStreamType(type);
if (name == nullptr)
{
throw std::invalid_argument("FrameMetadataTrackerOak: invalid StreamType value " +
std::to_string(static_cast<int>(type)));
}
m_stream_names.emplace_back(name);
}
}

const FrameMetadataOakTrackedT& FrameMetadataTrackerOak::get_stream_data(const ITrackerSession& session,
size_t stream_index) const
const FrameMetadataOakTrackedT& FrameMetadataTrackerOak::get_data(const ITrackerSession& session) const
{
return static_cast<const IFrameMetadataTrackerOakImpl&>(session.get_tracker_impl(*this)).get_stream_data(stream_index);
return static_cast<const IFrameMetadataTrackerOakImpl&>(session.get_tracker_impl(*this)).get_data();
}

} // namespace core
Loading
Loading