Skip to content

Add the Factory contact-rich assembly task on Newton - #6891

Merged
kellyguo11 merged 14 commits into
isaac-sim:developfrom
ooctipus:zhengyuz/factory-newton
Aug 14, 2026
Merged

Add the Factory contact-rich assembly task on Newton#6891
kellyguo11 merged 14 commits into
isaac-sim:developfrom
ooctipus:zhengyuz/factory-newton

Conversation

@ooctipus

@ooctipus ooctipus commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Description

Important

Depends on #7078 — that PR must be merged first. This branch is stacked on the effort-limit mirror of #6481, so the diff includes its single actuator commit. Review the Factory commits on top; after #7078 lands, rebasing will remove the parent commit from this diff.

Adds IsaacContrib-Factory-Franka, a preset-driven contact-rich assembly task under isaaclab_tasks.contrib.nist.

The task covers 12 NIST taskboard variants:

  • M16 nut threading
  • round peg insertion at 4, 8, 12, and 16 mm
  • rectangular rod insertion at 4, 8, 12, and 16 mm
  • small, medium, and large gear meshing

Assembly geometry, reset strategy, curriculum, robot, and physics are composed through presets rather than separate environment classes. The task supports explicit isaacsim_physx and newton_mjwarp presets, with physx retaining automatic backend selection.

Dependency

#7078 separates the two implicit-actuator effort fields:

  • effort_limit retains the actuator-facing rated force or torque.
  • effort_limit_sim controls the physics-solver clamp.

The Factory Franka configuration intentionally authors distinct rated and solver limits, so it cannot construct on unmodified develop.

Changes

  • Added the Factory task, assembly profiles, reset-state generation, sampling, rewards, observations, terminations, and curriculum.
  • Added the 12 task presets and RSL-RL configuration.
  • Added explicit Isaac Sim PhysX and Newton physics configuration.
  • Kept task assets on uploaded asset paths; no Factory-specific Nucleus root constant is introduced.
  • Bounded precollected reset states in the existing environment smoke harness so one- and two-environment CI cases remain smoke tests rather than building the full training bank.

Type of change

  • New feature (non-breaking)

Validation

  • uv run isaaclab -f
  • Preset decision suite: 19 passed
  • Exact Factory two-environment CI smoke case: 1 passed in 35.8 seconds
  • Five-iteration isaacsim_physx training run completed
  • Changelog fragment validation
  • No new test files were added while addressing the CI failures

@ooctipus
ooctipus requested a review from a team August 4, 2026 12:46
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Aug 4, 2026
@greptile-apps

greptile-apps Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a preset-driven Factory assembly task for Newton and supporting actuator, collision-schema, asset-path, and Newton cloning changes.

  • Adds nut-threading and peg-insertion scenes, MDP terms, reset strategies, sampling utilities, curricula, and RSL-RL configuration.
  • Preserves authored mesh collision approximations during Newton replication.
  • Separates implicit-actuator rated effort limits from solver-level clamps.

Confidence Score: 4/5

The PR should not merge until the accumulator constructs its collision analyzers correctly, because the Factory task otherwise fails during its first reset.

The active Factory reset configuration passes analyzer configuration objects into code that calls their class factory with the predicate-call signature, causing incompatible constructor arguments before any reset states can be collected.

Files Needing Attention: source/isaaclab_tasks/isaaclab_tasks/core/multi_task/utils/event_combinators.py and source/isaaclab_tasks/isaaclab_tasks/core/multi_task/reset_env_cfg.py

Important Files Changed

Filename Overview
source/isaaclab_tasks/isaaclab_tasks/core/multi_task/utils/event_combinators.py Adds state accumulation and reset-term composition, but the configured collision acceptance predicates are instantiated with incompatible arguments during precollection.
source/isaaclab_tasks/isaaclab_tasks/core/multi_task/reset_env_cfg.py Defines Factory reset chains and accumulator configuration, including the CollisionAnalyzerCfg values that reach the broken acceptance invocation.
source/isaaclab_tasks/isaaclab_tasks/core/multi_task/utils/sampling/impl/sampler_warp.py Adds Warp-backed scoring, probability calculation, and sampling for reset-state selection; no concrete failure was retained.
source/isaaclab_tasks/isaaclab_tasks/core/multi_task/factory_env_cfg.py Composes the manager-based Factory environment and preset-selected reset and curriculum behavior.
source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py Removes blanket post-import convex-hull approximation and delegates authored approximation handling to Newton's importer.
source/isaaclab/isaaclab/actuators/actuator_pd.py Resolves rated and solver-level implicit-actuator effort limits independently while retaining single-field compatibility.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Factory environment reset] --> B[ACCUMULATOR_RESET]
  B --> C[Run chained scene reset]
  C --> D[Evaluate collision acceptance conditions]
  D --> E[Collect accepted states]
  E --> F[Downsample state table]
  F --> G[Sample reset state per environment]
Loading

Reviews (1): Last reviewed commit: "Add skip fragments for the test-only bac..." | Re-trigger Greptile

Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/event_combinators.py Outdated

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The Factory task and supporting actuator/collision changes are substantial, but several merge-time issues remain: the task’s asset root is source-checkout-specific, public Newton mesh-simplification APIs are removed without deprecation, the lazy-export stub advertises an undefined symbol, and subset collision checks index packed SDF data incorrectly. There are also a broken type-only import and an avoidable dependency from generic multi-task utilities into the lift task.

  • Design and architecture: The preset-driven task structure and moving collision approximation authority from the Newton cloner to asset spawners are coherent. However, the domain-agnostic reset combinators directly depend on core.lift.mdp.events_cfg.SuccessMonitorCfg, coupling shared infrastructure to an unrelated task package. Move that shared configuration into the multi-task utility layer and update both consumers.
  • API: ImplicitActuator now preserves independent rated and solver effort limits while retaining the one-field fallback, and CollisionBaseCfg.mesh_collision_property exposes the existing schema dispatch. In contrast, NewtonCfg.simplify_meshes and related callable parameters are removed without the required prior deprecation. LOCAL_ASSET_PATH_DIR constructs a repository-only path that will not resolve in installed distributions, and the mdp lazy-export stub lists an undefined RESET_STRATEGIES symbol.
  • Implementation: The SDF collision analyzer packs collider slots and counts using the global environment count and absolute environment IDs, but the query kernel indexes them using the queried subset size and local subset index. Collision validation therefore reads incorrect environments for partial or reordered env_ids; the kernel must use the global stride and absolute ID. The observations.py type-only Offset import also targets a nonexistent mdp.util module and should reference multi_task.utils.pose_offset.

Significant concerns. Posted 6 actionable findings inline.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab/isaaclab/utils/assets.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/__init__.pyi Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/sdf.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/observations.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/event_combinators.py Outdated
@ooctipus
ooctipus force-pushed the zhengyuz/factory-newton branch from d297e60 to 6a1dd6d Compare August 4, 2026 12:52
@ooctipus
ooctipus force-pushed the zhengyuz/factory-newton branch 2 times, most recently from 1295350 to 3eb84ef Compare August 4, 2026 20:43
@kellyguo11 kellyguo11 moved this to In review in Isaac Lab Aug 5, 2026
@ooctipus
ooctipus force-pushed the zhengyuz/factory-newton branch 4 times, most recently from f28550c to 7e6e97a Compare August 6, 2026 10:23

@AntoineRichard AntoineRichard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-generated review

This PR is not ready to merge. The latest head still has functional blockers in task startup and collision validation, and it carries substantial unused/copied infrastructure that should be removed before this becomes a showroom task.

Merge blockers

  • The task hard-codes NVIDIA's internal isaac-dev Nucleus, so external users cannot load its assets.
  • SuccessMonitorCfg is imported from core/lift/mdp/events_cfg.py, which has been deleted on current develop; the merged tree will fail to import this task.
  • Reset precollection calls CollisionAnalyzer with the predicate signature instead of constructing it from its config.
  • The SDF kernel addresses globally packed collider data with subset-local strides/indices.
  • The process-global collider cache retains stage-owned prims and an old environment count across environment reconstruction.

The prior automated comments about the analyzer constructor, SDF subset indexing, nonexistent .util.pose_offset import, undefined RESET_STRATEGIES, and lift-layer coupling remain valid at this head.

Required PR reduction and consolidation

  • Remove/defer the 514-line Warp sampler and the Frontier strategy: no production config in this PR enables warp=True or uses Frontier; only their tests do. Keep the Torch Beta/Uniform path used by Factory.
  • Remove the terrain/multi-task command observations and unused reward/termination terms copied into contrib.nist; they are not referenced by the Factory config and create a false public surface.
  • Generate the repeated socket/plug asset configs through a small typed helper, consolidate identical keypoint/preset classes, and delete commented-out alternatives.
  • Fold the duplicated Hydra import test into the existing shared test and remove config-only smoke/import tests already covered by the shared Factory runtime test and other unit tests. Keep the two scheduler behavior tests in the curriculum test module.
  • Add focused regressions for reordered/subset env_ids and for analyzer reconstruction across stages/environment counts. Those tests catch the concrete failures; many current tests only assert trivial properties or implementation strings.

The comments/docstrings also need a full editorial pass: keep them functional and durable, remove author tags/debugging narratives/value-selection postmortems, and delete stale references to terrain, connectors, old layouts, and “backwards compatibility” for new symbols.

I did not run project code, pre-commit, or GPU tests locally because benchmarks are running on this machine. Static git diff --check passed. GitHub CI at this head is currently blocked with failures in two isaaclab_tasks shards, docs, isaaclab_rl, and visualizers.

Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/factory_assets_cfg.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/reset_env_cfg.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/event_combinators.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/sdf.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/rigid_object_hasher.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/events.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/observations.py Outdated
Comment thread source/isaaclab_tasks/test/test_factory_hydra_imports.py Outdated
Comment thread source/isaaclab_tasks/changelog.d/zhengyuz-factory-newton.rst
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/config/agents/rsl_rl_ppo_cfg.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/config/__init__.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/config/__init__.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/config/franka_factory_cfg.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/config/franka_factory_cfg.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/events.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/events.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/mdp/observations.py Outdated
Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/nist/utils/mesh_ops.py
@ooctipus
ooctipus force-pushed the zhengyuz/factory-newton branch from 4ac941d to 2f7dda8 Compare August 13, 2026 06:31
@ooctipus
ooctipus force-pushed the zhengyuz/factory-newton branch 2 times, most recently from d3bd89e to 8c66461 Compare August 13, 2026 17:45
Treat effort_limit as the actuator-facing rated force or torque and effort_limit_sim as the physics-solver clamp. Preserve existing one-field fallbacks while allowing both fields to carry distinct values, mirroring the velocity-limit semantics introduced in isaac-sim#6481.
Dropping the cloner's blanket approximation pass leaves every collider at
its USD-authored approximation, and USD defaults that to none. A sweep of
all 132 registered tasks, comparing each collider's resolved Newton shape
before and after, found three tasks whose colliders changed: the lift
multi-asset object and the reorient dex cube, both spawned as tessellated
mesh primitives with no approximation authored.

Author convexHull on their shared collision props, which the multi-asset
spawner propagates to every variant, restoring the shapes those tasks
resolved before.
The Lift object preset has two variants: a multi-asset spawner of Mesh*
primitives, and a plain CuboidCfg that spawns an analytic UsdGeom.Cube.
Only the former is a mesh, so only the former was ever hulled by the
cloner -- authoring a mesh-collision approximation on the cube claims a
property the prim cannot have.

The collider sweep confirms it was doing nothing: the tasks the hull is
meant to protect resolve identical shapes without it.
Add IsaacContrib-Factory-Franka, a manager-based assembly task covering
NIST taskboard M16 nut-threading and round-peg insertion with a Franka
Panda. Assembly, robot, reset strategy and curriculum are each a
PresetCfg, so a run is selected with
presets=franka,nut_thread_m16,newton_mjwarp instead of a bespoke env
class per assembly.
The task registers one environment, IsaacContrib-Factory-Franka, and one
robot. Registering that robot as a `franka` variant meant every command
had to repeat a selector the task name already implies.

Bind the robot-specific fields to each preset's `default` instead, so a
run selects only what actually varies -- the assembly and the physics
backend. Adding a second robot still works the same way: give it a named
variant alongside the default.
Every scene variant pinned num_envs=2, so a training run that forgot
--num_envs looked healthy while collecting almost no experience. Set the
shared scene base to 4096, matching the other manager-based tasks, and
let the variants inherit it -- they only need to differ in env_spacing.

play resolves the same config and applies no cap of its own, so play_mode
takes it back down to something you can actually watch.
The task is a contributed task family, so it belongs under contrib
rather than core. Nothing in core imported it, so the dependency
direction stated in isaaclab_tasks (core never depends on contrib) is
preserved.

Name the package after the NIST taskboard rather than the task: contrib
already ships an unrelated factory package, and the old multi_task name
described the directory's history rather than its contents. The gym id
IsaacContrib-Factory-Franka is unchanged.
The gear variants put two gears on the board besides the pair being
assembled, and neither the accumulator nor the reset acceptance gate knew
about them.

The bank restored only nistboard, fixed_asset, held_asset and robot, so a
replayed state left the spare gears wherever the previous episode ended
while the board moved underneath them. The held asset was screened for
collisions against the board, table, robot and fixed asset but not against
those gears, so a state seating it inside one was banked as valid. PhysX
pushes such a state apart; MuJoCo/Newton diverges to NaN a few steps later,
which is why only the Newton runs died.

Scope both lists per assembly so the spare gears are restored and screened.
@ooctipus
ooctipus force-pushed the zhengyuz/factory-newton branch from 8c66461 to fddc02f Compare August 13, 2026 18:08
@ooctipus

Copy link
Copy Markdown
Collaborator Author

Extracted the implicit-actuator effort-limit semantics into #7078, the effort-limit mirror of #6481, and rebased this branch on that single prerequisite commit. The direct diff from #7078 to this head now touches only isaaclab_tasks.

Revalidated after the split:

  • preset decision suite: 19 passed
  • exact IsaacContrib-Factory-Franka-2-cuda smoke case: 1 passed in 35.8 seconds
  • repository formatter and changelog checks: passed

@kellyguo11
kellyguo11 merged commit 50ac886 into isaac-sim:develop Aug 14, 2026
80 of 83 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in Isaac Lab Aug 14, 2026
@ooctipus
ooctipus deleted the zhengyuz/factory-newton branch August 14, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants