Skip to content

Add topological modeling for action forward dynamics - #223

Open
teerthsharma wants to merge 1 commit into
NVIDIA:mainfrom
teerthsharma:codex/topological-modeling-action-fd
Open

Add topological modeling for action forward dynamics#223
teerthsharma wants to merge 1 commit into
NVIDIA:mainfrom
teerthsharma:codex/topological-modeling-action-fd

Conversation

@teerthsharma

Copy link
Copy Markdown

Summary

This PR adds Topological Modeling for Cosmos Action Forward Dynamics as one cohesive optional evaluation layer for Cosmos3 action FD rollouts.

It covers the full modeling scope:

  • FD/FDTD modeling: adds finite-difference topology traces over rollout state so frame/chunk dynamics have deterministic velocity, acceleration, local-change, and convergence diagnostics.
  • Topology metrics: evaluates visual FD rollouts from binary or labeled masks with connected components, hole/loop proxies, Euler-characteristic-style summaries, drift, stability, JSON, and CSV outputs.
  • Topological inference: adds compact topology state vectors, weighted sparse-neighborhood distance, Betti-stability windows, and theorem-style convergence gates.
  • Swarm robotics extension: adds sparse top-k specialist ranking using topology-aware state similarity, task tags, reliability, cost, and utilization signals.

The helper is side-effect-free. It does not alter default Cosmos3 inference and does not require GPU/model weights/segmentation dependencies for CI-style validation.

Files

  • cookbooks/cosmos3/generator/action/topology_helpers.py
    • mask topology metrics
    • optional persistent-homology enrichment when ripser is available
    • finite-difference rollout traces
    • convergence gates
    • topology state vectors
    • sparse specialist ranking
    • JSON/CSV report writers
  • cookbooks/cosmos3/generator/action/topological_modeling.md
    • reviewer-facing modeling note for FD/FDTD, topology metrics, topological inference, and swarm robotics extension
  • cookbooks/cosmos3/generator/action/README.md
    • cookbook entry point and minimal mask-first usage snippet
  • tests/cookbooks/cosmos3/generator/action/test_topology_helpers.py
    • 16 CPU-only synthetic-mask tests

Verification

python -m py_compile cookbooks/cosmos3/generator/action/topology_helpers.py
python -m unittest tests.cookbooks.cosmos3.generator.action.test_topology_helpers -v
git diff --check

All passed locally.

Maintainer context

Opened proposal issue before implementation: #222

Non-goals

  • No change to default Cosmos3 generation behavior.
  • No required segmentation model dependency.
  • No GPU or model-weight requirement for tests.
  • No claim of improved generation quality without a controlled validation protocol.
  • No vendored external research prototype code.

@fwd4

fwd4 commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

heads up for reviewers: NVIDIA/NeMo-Relay#282

@mli0603

mli0603 commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Interesting feature. @teerthsharma is the idea that if FD retains the same connected components, the prediction is good? And this is used as a proxy for FD eval?

@teerthsharma

Copy link
Copy Markdown
Author

Interesting feature. @teerthsharma is the idea that if FD retains the same connected components, the prediction is good? And this is used as a proxy for FD eval?

Same connected components by topology yes !!
The prediction seems good in my end

Cosmos3 action forward-dynamics rollouts are inspected frame by frame, so
failures that live in structure rather than pixels -- an object mask splitting
across an autoregressive chunk boundary, a contact loop opening, a stable
component collapsing -- do not surface in pixel error. This adds an optional,
side-effect-free evaluation layer over rollout masks that reports connected
components, enclosed-background hole proxies, Euler characteristic, drift,
finite-difference topology dynamics, and a convergence certificate.

Default Cosmos3 generation behavior, model weights, GPU requirements, and the
dependency set are unchanged. The core path is pure Python over lists; ripser is
optional and off by default.

Files:

- cookbooks/cosmos3/generator/action/topology_helpers.py
- cookbooks/cosmos3/generator/action/topological_modeling.md
- cookbooks/cosmos3/generator/action/README.md
- tests/cookbooks/cosmos3/generator/action/test_topology_helpers.py

Determinism

The report is a comparison artifact, so its bytes are fixed by the input alone.
Four sources of variance are closed:

- Labels are evaluated in sorted order. Previously `{"object": ..., "gripper":
  ...}` and `{"gripper": ..., "object": ...}` produced reports whose frame list,
  CSV row order, and finite-difference trace differed while the data did not.
- `rank_topology_specialists` breaks score ties on `specialist_id`. Previously a
  tie resolved to whichever candidate the caller listed first.
- Persistent homology is either explicitly absent or explicitly required.
  `PersistentHomologyConfig` drops the `enabled` flag, which silently did
  nothing while `backend` defaulted to `"none"`; `backend="ripser"` now raises
  `ImportError` rather than degrading to a different set of numbers on a machine
  without the package.
- `RolloutSpec` rejects a non-positive `fps`. `fps=-15` clamped the
  finite-difference `dt` to `1e-9` and reported a topology speed of 2.08e9
  against a gate threshold of 2.0.

Digital topology

Two conditions make `components - holes` an Euler characteristic rather than two
unrelated counts, and `TopologyConfig` now enforces both.

Complementary connectivity: `foreground_connectivity` and
`background_connectivity` must be 8/4 or 4/8, the digital Jordan curve
condition. On the diagonal ring `.#. / #.# / .#.`, the 8/8 pair reports zero
holes because the enclosed pixel reaches the border diagonally, and the 4/4 pair
splits the ring into four components and reports chi = 3. Only the complementary
pairs return chi = 0. A matching pair raises `ValueError`.

One filtered foreground: `min_component_area_px` is applied before holes are
counted. A ring of area 8 under a threshold of 16 previously reported zero
components and one hole, chi = -1 for an empty foreground; it now contributes
neither.

Persistent homology

`ripser` is called with an `(n, 2)` float array. The previous call passed a list
of pixel tuples and raised `'list' object has no attribute 'shape'`, so the
enrichment path failed in every environment where ripser was installed and was
reported as status `"failed"`. A skipped-unless-installed annulus test covers it.

`PersistentHomologySummary.betti_0`/`betti_1` are renamed `h0_bars`/`h1_bars`.
They are diagram cardinalities, not Betti numbers: a Rips H0 diagram carries one
bar per input point, so `h0_bars` always equals `sampled_points`. The Betti
proxies the module stands behind remain the combinatorial `components` and
`holes`.

Validation

41/41 unittest cases pass from this commit on Python 3.11.9, Windows 11, with
numpy 1.26.4 and ripser 0.6.14 present so the optional path is exercised
rather than skipped. The suite grows from 16 cases to 41. Run unchanged against the prior
helper on the same interpreter, it reports 13 failures and 1 error out of 41, so
every determinism, digital-topology, persistent-homology, and metric-axiom claim
above is pinned by a case that fails without the corresponding fix.

Ground-truth coverage asserts the value a correct implementation must return:
solid disk chi = 1, annulus chi = 0, figure-eight chi = -1, six separated blobs
chi = 6, diagonal ring chi = 0, and a scattered-points negative control that
must report zero holes. `topology_state_distance` is checked against the four
metric axioms over 4 vectors: identity, positivity on 6 distinct pairs, symmetry
on 6 pairs, and the triangle inequality over all 64 triples.

Cost

A 16-frame single-label rollout, one run per size on the machine above, prior
helper to this commit: 64x64 45ms to 30ms, 128x128 187ms to 128ms,
256x256 734ms to 541ms, 480x640 4.296s to 3.007s. The reduction comes from
hoisting the neighbor-offset table out of the flood-fill inner loop, where
cProfile attributed 0.35s of 2.43s at 256x256 to 1,048,576 rebuild calls, and
from converting each mask to a boolean grid once per frame instead of four
times. Complexity is unchanged at O(pixels) per frame per pass.

Limits

The stability score is a hand-tuned heuristic in [0, 1], not a calibrated
quantity, and is marked as such in the source; its constants should be fit
against labelled rollouts before scores are compared across runs. The
convergence gate thresholds are defaults, not derived bounds. A rollout shorter
than `betti_window` yields no windows and scores 1.0 vacuously; `betti_windows`
is reported in the metrics so that case is distinguishable, but the check itself
still passes. Timings are single-run on one machine with no confidence interval
and no cross-platform control. The metric is diagnostic and should not be
reported as a generation improvement without a controlled validation protocol.

Relates to NVIDIA#222
@teerthsharma
teerthsharma force-pushed the codex/topological-modeling-action-fd branch from 94b8229 to 73e97ed Compare August 11, 2026 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants