-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Restore remote asset URLs in stage dumps #7080
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| Added | ||
| ^^^^^ | ||
|
|
||
| * Added :func:`~isaaclab.utils.assets.unmirror_file_path`, which maps a locally cached asset copy | ||
| written by :func:`~isaaclab.utils.assets.retrieve_file_path` back to the URL it was downloaded | ||
| from. Exports of a stage that references cached copies can use it to name the source assets | ||
| instead of machine-specific cache paths. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -118,6 +118,11 @@ def _resolve_asset_root() -> str: | |
|
|
||
| _GIT_SSH_RE = re.compile(r"^[^@/:]+@[^:]+:.+") | ||
|
|
||
| _MIRROR_URL_SCHEMES = frozenset({"http", "https", "omniverse"}) | ||
| """URL schemes whose assets are cached locally, and which therefore start a cache layout.""" | ||
|
|
||
| _MIRROR_NETLOC_PORT_RE = re.compile(r"^(.+)_(\d+)$") | ||
|
|
||
|
|
||
| def retrieve_git_asset_path( | ||
| git_path: str, local_path: str, cache_dir: str | None = None, force_update: bool = False | ||
|
|
@@ -305,6 +310,33 @@ def _mirror_path(url: str, download_dir: str) -> str: | |
| return os.path.join(download_dir, parsed.scheme, netloc, *parsed.path.lstrip("/").split("/")) | ||
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. additional problem I found with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. tests should be added for Windows paths like this |
||
|
|
||
| def unmirror_file_path(path: str) -> str: | ||
| """Reverses :func:`retrieve_file_path` caching, mapping a cached copy back to its source URL. | ||
|
|
||
| A remote asset is cached under ``<download_dir>/<scheme>/<host>/<path>``, and stages reference | ||
| that cached copy rather than the URL it came from. Exports of such a stage therefore carry | ||
| absolute paths that only resolve on the machine holding the cache. This recovers the URL so an | ||
| export can name the source asset instead. | ||
|
|
||
| Args: | ||
| path: Local filesystem path, typically an asset path read from a USD layer. | ||
|
|
||
| Returns: | ||
| The URL the path was cached from, or ``""`` when it does not lie inside a cache layout. | ||
| """ | ||
| parts = path.replace(os.sep, "/").split("/") | ||
| # the last two components are the host and at least one path component, so a scheme found | ||
| # there cannot be the start of a cache layout | ||
| for index, part in enumerate(parts[:-2]): | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Warning · Design Architecture — Cache detection not anchored to download directory The scan accepts any path component named |
||
| if part.lower() not in _MIRROR_URL_SCHEMES: | ||
| continue | ||
| netloc, *remainder = parts[index + 1 :] | ||
| # ``_mirror_path`` writes a port separator as '_', which is not valid in a host name | ||
| netloc = _MIRROR_NETLOC_PORT_RE.sub(r"\1:\2", netloc) | ||
| return f"{part.lower()}://{netloc}/{'/'.join(remainder)}" | ||
|
Comment on lines
+330
to
+336
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When a locally authored asset path contains a directory named Knowledge Base Used: Terrains and Shared Utilities |
||
| return "" | ||
|
|
||
|
|
||
| def _remote_fingerprint(url: str) -> dict | None: | ||
| """Provider metadata identifying the revision of ``url`` the server currently holds. | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Rendering test stage dumps rewrite cached asset paths back to their source URLs, so a stage saved | ||
| through ``ISAAC_LAB_SAVE_STAGES`` no longer carries texture paths that only resolve on the machine | ||
| that ran the test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should these schemes be hard coded. I think I may have seen other schemes as well like
s3://There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This sounds like a question, but I am not sure. Are you suggesting that we shouldn't hard code those and rely on other functionality? Have you had something specific in mind?