[Test] Cover contributed environments in the shared smoke test - #7068
[Test] Cover contributed environments in the shared smoke test#7068StafaH wants to merge 11 commits into
Conversation
Greptile SummaryThe PR consolidates core environment smoke coverage into a shared three-backend matrix, moves eligible contributed environments into that test module, removes obsolete Play/Newton discovery filters, and adds explicit garbage collection before simulation teardown.
Confidence Score: 3/5This PR is not safe to merge until the incomplete contributed test and preset-incompatible core matrix are corrected. The consolidated module cannot be imported because its final test function lacks a body, and after that is repaired the unconditional preset matrix reaches core task configurations that do not define the requested backends. Files Needing Attention: source/isaaclab_tasks/test/core/test_environments.py, source/isaaclab_tasks/test/env_test_utils.py Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
Registry[Gym task registry] --> Filter[setup_environment filters]
Filter --> Core[Core task list]
Filter --> Contrib[Contrib task list]
Core --> Matrix[Three physics presets]
Matrix --> Runner[_run_environments]
Contrib --> ContribRunner[Default-backend rollout]
Runner --> Teardown[Close env and collect garbage]
ContribRunner --> Teardown
Teardown --> Clear[Clear SimulationContext]
Reviews (1): Last reviewed commit: "Revert "Update preset tests for Newton d..." | Re-trigger Greptile |
| ), | ||
| ) | ||
| @pytest.mark.isaacsim_ci | ||
| def test_contrib_environments(task_name, num_envs, device): |
There was a problem hiding this comment.
When pytest imports this module, the empty test_contrib_environments definition raises an IndentationError, preventing collection of both the core backend matrix and the contributed smoke test.
| def test_contrib_environments(task_name, num_envs, device): | |
| def test_contrib_environments(task_name, num_envs, device): | |
| # run a short rollout with random actions to verify the environment loads and steps | |
| _run_environments(task_name, device, num_envs, create_stage_in_memory=False) |
| # Local imports should be imported last | ||
| from env_test_utils import _run_environments, setup_environment # isort: skip | ||
|
|
||
| @pytest.mark.parametrize("physics_preset_name", ["newton_mjwarp", "physx", "isaacsim_physx"]) |
There was a problem hiding this comment.
There was a problem hiding this comment.
Isaac Lab Review Bot
The consolidated smoke coverage and three-preset core matrix are coherent, but the moved contributed-environment test no longer skips IsaacContrib-Franka-Pour, despite its documented external reset-dataset requirement.
- Design and architecture: Consolidating core and eligible contributed smoke coverage into the shared environment test is consistent with the PR’s stated design. The preset axis replaces the dedicated Newton test while contributed tasks continue to use their configured default backend.
- API: All changed callers of the test-local
setup_environmenthelper were updated consistently after removinginclude_playandnewton_mjwarp_envs. No unsupported public API change is evident. - Implementation: The teardown collection occurs before
SimulationContext.clear_instance(), preserving the intended Warp finalizer ordering. However, the replacement contributed test parametrizesIsaacContrib-Franka-Pourwithout carrying over its artifact-dependent skip or excluding it during selection, so environments without that external dataset will now run a known-ineligible case.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
| ), | ||
| ) | ||
| @pytest.mark.isaacsim_ci | ||
| def test_contrib_environments(task_name, num_envs, device): |
There was a problem hiding this comment.
🟡 Warning · Implementation — Contrib move drops Franka-Pour artifact skip
The deleted test_contrib_environments_smoke.py skipped IsaacContrib-Franka-Pour because it requires an external reset-dataset artifact. test_contrib_environments reuses the same setup_environment filters (none of which exclude that task) but omits the skip, and XFAIL_TASKS is empty, so the task is now parametrized and will fail wherever the dataset is absent. Re-add the pytest.skip guard or exclude the task during selection.
…-for-tests [Test] Generate Franka Pour resets for contrib smoke test
Adds contributed task coverage to the consolidated environment smoke test.\n\n- Runs eligible contrib tasks through each task’s configured default backend only.\n- Keeps dedicated Factory, Showcase, PickPlace/Stack, Teleop, AutoMate, and Skillgen suites separate.\n- Includes the shared core three-backend matrix, obsolete Play/MJWarp-discovery utility cleanup, and teardown reliability cleanup.\n\nThe shared test commit is intentionally identical to the Newton-default shards; rebase/drop it after the first shard merges.\n\nExtracted from #6980.