From 0bca837d84944479288cd853358316df11be6292 Mon Sep 17 00:00:00 2001 From: espg Date: Mon, 24 Aug 2026 15:48:00 -0700 Subject: [PATCH 1/6] phase 1 of issue #176 --- docs/api/convert.md | 12 ++++-------- docs/api/coverage.md | 7 +++++++ docs/api/moc_object.md | 1 + docs/api/rank_xy.md | 19 +++++++++++++++++++ docs/api/toc_object.md | 1 + docs/arrow_interchange.md | 25 +++++++++++++++++++++++++ mkdocs.yml | 1 + 7 files changed, 58 insertions(+), 8 deletions(-) create mode 100644 docs/api/rank_xy.md diff --git a/docs/api/convert.md b/docs/api/convert.md index fbbe7543..67a6a3a4 100644 --- a/docs/api/convert.md +++ b/docs/api/convert.md @@ -26,13 +26,9 @@ latitude→latitude mapping on its own. - mort2healpix - mort2norm - norm2mort + - geo2uniq + - norm2uniq + - uniq2geo + - unique2parent - geodetic_to_authalic - authalic_to_geodetic - -!!! note "Not yet documented here" - - The UNIQ helpers (`geo2uniq`, `norm2uniq`, `uniq2geo`, `unique2parent`) are - omitted while their signatures are in flux — see - [issue #136](https://github.com/espg/mortie/issues/136). `heal_norm` is - omitted because it is being removed under - [PR #130](https://github.com/espg/mortie/pull/130). diff --git a/docs/api/coverage.md b/docs/api/coverage.md index 00d689ff..9fea2622 100644 --- a/docs/api/coverage.md +++ b/docs/api/coverage.md @@ -8,7 +8,14 @@ public entry point, `polygons_to_morton_mocs`, lives in multipart form is reached through `from_geometry` / `from_wkb` / `from_wkt` with `moc=True`, or `mortie.Moc`). +The ring-validity checks below report whether any documented winding +convention is in play for a ring *before* covering it — see +[Ring validity](../coverage_methods.md#ring-validity) for the narrative. + ::: mortie.coverage options: members: - morton_coverage + - ring_validity + - ring_is_simple + - RingValidity diff --git a/docs/api/moc_object.md b/docs/api/moc_object.md index dfeb1abd..5397e0c3 100644 --- a/docs/api/moc_object.md +++ b/docs/api/moc_object.md @@ -76,3 +76,4 @@ answer can err near a boundary. options: members: - Moc + - moc diff --git a/docs/api/rank_xy.md b/docs/api/rank_xy.md new file mode 100644 index 00000000..7d3ee5f6 --- /dev/null +++ b/docs/api/rank_xy.md @@ -0,0 +1,19 @@ +# mortie.rank_xy + +Subtree-local rank <-> face-local `(x, y)` bit deinterleave for 2-D block +views (issue #149). A depth-`d` subtree holds `4**d` cells whose ascending +packed-word order is a Z-order (morton) curve over a `2**d x 2**d` block; +`rank_to_xy` / `xy_to_rank` convert between a cell's **rank** in that block +and the deinterleaved pair, matching the healpy / HEALPix C++ `pix2xyf` +convention (origin at the subtree's south corner). The input is rank-space, +**not** packed morton words — strip the shard prefix down to the base-4 +digit-tail rank first. Normative statement: +[specification.md §8](../specification.md#8-rank-space-x-y-deinterleave); +the public functions ship the Rust kernel (`src_rust/src/rank_xy.rs`). The +names stay flat on the package (`mortie.rank_to_xy`, `mortie.xy_to_rank`). + +::: mortie.rank_xy + options: + members: + - rank_to_xy + - xy_to_rank diff --git a/docs/api/toc_object.md b/docs/api/toc_object.md index a38a3b76..99d62e02 100644 --- a/docs/api/toc_object.md +++ b/docs/api/toc_object.md @@ -55,3 +55,4 @@ each answer can err near a span edge (the quanta are ~2–4 s). options: members: - Toc + - toc diff --git a/docs/arrow_interchange.md b/docs/arrow_interchange.md index 30eb4489..44459e54 100644 --- a/docs/arrow_interchange.md +++ b/docs/arrow_interchange.md @@ -17,6 +17,31 @@ workers (e.g. an AWS Lambda layer without pyarrow). The raw Arrow C structs are built in Rust (via the `arrow` crate), so nothing on the critical path imports pyarrow. +## The pyarrow extension classes: `MortonIndexType` / `MortonIndexExtArray` + +The pyarrow skin's two classes are public as `mortie.MortonIndexType` and +`mortie.MortonIndexExtArray` (and on `mortie.arrow`), but they are **built +lazily on first attribute access** behind a module `__getattr__`, so that +importing mortie never imports pyarrow. That is also why they have no +rendered [API page](api/arrow.md): mkdocstrings resolves modules statically, +and the classes do not exist until first touched. They are documented here +instead. Touching either name without pyarrow installed raises an +`ImportError` pointing at the missing extra. + +- **`MortonIndexType`** is the `pyarrow.ExtensionType` subclass over + `uint64` storage with extension name `mortie.morton_index`. It carries no + parameters — its serialized form is empty; the extension name is the whole + identity — so the type survives parquet / IPC round-trips. + `morton_index_type()` builds, registers, and returns the singleton + instance; there is no reason to construct the class directly. +- **`MortonIndexExtArray`** is the matching `pyarrow.ExtensionArray` + subclass: what `from_morton_index` returns, and what pyarrow hands back + when the registered type resolves on read. Its one addition over the + stock class is `to_numpy(**kwargs)`, which materializes the `uint64` + storage (defaulting `zero_copy_only=False` so a null-bearing array + converts); for the null → sentinel-`0` word mapping, go through + `to_morton_index` instead. + ## Producing a column (any Arrow lib) `export_c_array` returns the `(schema_capsule, array_capsule)` pair of the diff --git a/mkdocs.yml b/mkdocs.yml index 075b89e0..f4a7a784 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -47,6 +47,7 @@ nav: - API reference: - convert: api/convert.md - orders: api/orders.md + - rank_xy: api/rank_xy.md - buffer: api/buffer.md - coverage: api/coverage.md - moc kernel: api/moc.md From 044b03af82fc2c1b4299a7b34d5738e6eaad0285 Mon Sep 17 00:00:00 2001 From: espg Date: Mon, 24 Aug 2026 15:58:01 -0700 Subject: [PATCH 2/6] phase 2 of issue #176 --- mortie/tests/test_arrow.py | 8 ++ mortie/tests/test_docs_api_pages.py | 137 ++++++++++++++++++++++++++-- 2 files changed, 139 insertions(+), 6 deletions(-) diff --git a/mortie/tests/test_arrow.py b/mortie/tests/test_arrow.py index f4eb7db1..fe2fc0da 100644 --- a/mortie/tests/test_arrow.py +++ b/mortie/tests/test_arrow.py @@ -290,6 +290,14 @@ def _blocked(name, *args, **kwargs): raise ImportError("blocked for test") return real_import(name, *args, **kwargs) + import mortie + + # The fresh import below also rebinds the parent package's ``arrow`` + # attribute to the transient copy, and ``delitem`` only saves the + # ``sys.modules`` entry -- register the attribute too so ``undo()`` + # restores both, keeping ``mortie.arrow`` identical to + # ``sys.modules["mortie.arrow"]`` (the docs pin checks by identity). + monkeypatch.setattr(mortie, "arrow", sys.modules["mortie.arrow"]) monkeypatch.delitem(sys.modules, "mortie.arrow", raising=False) for mod in list(sys.modules): if mod == "pyarrow" or mod.startswith("pyarrow."): diff --git a/mortie/tests/test_docs_api_pages.py b/mortie/tests/test_docs_api_pages.py index d26d705a..1d72e8d8 100644 --- a/mortie/tests/test_docs_api_pages.py +++ b/mortie/tests/test_docs_api_pages.py @@ -8,15 +8,25 @@ **stale-entry** direction: every ``members:`` name in ``docs/api/*.md`` must resolve as an attribute of the module its page documents. -The reverse direction is deliberately not pinned: nothing here requires every -public name to appear on *some* page, so a move that deletes a member from its -old page and forgets to add it to the new one still vanishes silently — as do -the public names that have no ``docs/api`` entry today. Closing that would -mean pinning an explicit roster of undocumented names, a docs-policy decision -rather than a drift check. +The reverse direction is pinned too (issue #176): every non-submodule name in +``mortie.__all__`` must appear on exactly one page — keyed on *(defining +module, name)*, so the arrow skins of core functions stay legal on their own +page — and every submodule in ``__all__`` must have a page whose ``:::`` block +is that module. A move that deletes a member from its old page and forgets +the new one now fails here instead of vanishing silently. The only names +allowed off the pages are the two lazily-built Arrow classes +(``LAZY_ARROW_UNDOCUMENTED``), which mkdocstrings cannot resolve statically. + +And the docs cannot outgrow the frozen surface either: every ``members:`` +entry must be in ``__all__`` by name, or sit in ``MODULE_SCOPED_DOCUMENTED`` +— the documented names deliberately reached through their submodule +(``mortie.arrow.export_c_array``, ``mortie.morton_index.MortonIndexScalar``) +rather than flat. Growing either roster is a deliberate act reviewed here, +not a silent omission. """ import importlib +import inspect import re from pathlib import Path @@ -24,6 +34,38 @@ API_DIR = Path(__file__).resolve().parents[2] / "docs" / "api" +# The two pyarrow extension classes are defined inside ``_build_type()`` and +# reached only through ``arrow.__getattr__`` (pyarrow is an optional extra), +# so griffe's static resolution cannot see them and no ``members:`` entry can +# render them. They are documented narratively in docs/arrow_interchange.md +# instead — the structural reason will not expire, so this roster should +# never grow (issue #176). Never ``getattr`` these here: resolving them +# raises ImportError when pyarrow is absent. +LAZY_ARROW_UNDOCUMENTED = frozenset({"MortonIndexType", "MortonIndexExtArray"}) + +# Documented names deliberately *not* flat on the package: each is public as +# an attribute of a submodule that is itself in ``__all__``. The C Data +# Interface trio is namespaced interop plumbing (``mortie.arrow.export_c_array``, +# issue #93); ``MortonIndexScalar`` is the repr/scalar type handed back by the +# ExtensionArray, spelled ``mortie.morton_index.MortonIndexScalar`` (#104). +MODULE_SCOPED_DOCUMENTED = frozenset({ + ("mortie.arrow", "export_c_array"), + ("mortie.arrow", "export_c_schema"), + ("mortie.arrow", "import_c_array"), + ("mortie.morton_index", "MortonIndexScalar"), +}) + +_MISSING = object() + + +def all_pages(): + """Map each documented module to its page name and ``members:`` roster.""" + return { + module: (page.name, members) + for page in sorted(API_DIR.glob("*.md")) + for module, members in [page_members(page)] + } + def page_members(path): """The (module, members) a docs/api page declares, from its mkdocstrings block.""" @@ -55,3 +97,86 @@ def test_every_member_resolves_on_its_module(page): "(mkdocstrings drops these silently — the rendered page just loses them)" ) assert len(members) == len(set(members)), f"{page.name}: duplicate members" + + +def test_every_public_name_documented_on_exactly_one_page(): + # The reverse direction of the pin above (issue #176): a name in + # ``mortie.__all__`` that no page lists has no rendered API entry at all + # — deleting ``- morton_buffer`` from buffer.md must fail here, not + # vanish silently. Keyed on (defining module, name) by object identity, + # so a page documenting another module's *skin* of the same name (the + # arrow forms of from_wkb / polygons_to_morton_mocs) neither satisfies + # nor double-counts the flat name. + import mortie + + pages = all_pages() + problems = [] + for name in mortie.__all__: + if name in LAZY_ARROW_UNDOCUMENTED: + continue # structurally unrenderable; see the roster's comment + obj = getattr(mortie, name) + if inspect.ismodule(obj): + if f"mortie.{name}" not in pages: + problems.append(f"submodule {name}: no docs/api page") + continue + homes = [ + fname + for module, (fname, members) in pages.items() + if name in members + and getattr(importlib.import_module(module), name, _MISSING) is obj + ] + if len(homes) != 1: + problems.append(f"{name}: on {homes or 'no page'}") + assert not problems, ( + "public names must render on exactly one docs/api page " + f"(the page of the module they are bound from): {problems}" + ) + + +def test_lazy_arrow_roster_is_not_stale(): + # If someone finds a way to render the lazy classes (stub declarations, + # a griffe extension), the allowlist must shrink in the same change. + import mortie + + documented = {name for _, (_, members) in all_pages().items() for name in members} + assert not LAZY_ARROW_UNDOCUMENTED & documented, ( + "allowlisted-as-undocumentable names now appear on a page — prune " + f"LAZY_ARROW_UNDOCUMENTED: {sorted(LAZY_ARROW_UNDOCUMENTED & documented)}" + ) + missing = LAZY_ARROW_UNDOCUMENTED - set(mortie.__all__) + assert not missing, f"allowlisted names no longer public: {sorted(missing)}" + + +def test_every_documented_name_is_public(): + # The docs cannot outgrow the frozen surface: a ``members:`` entry must + # be reachable from ``mortie.__all__`` — flat by name, or through the + # justified module-scoped roster. Growing MODULE_SCOPED_DOCUMENTED is a + # deliberate, reviewed act (issue #176). + import mortie + + pages = all_pages() + public = set(mortie.__all__) + stray = [ + f"{fname}: {name}" + for module, (fname, members) in pages.items() + for name in members + if name not in public and (module, name) not in MODULE_SCOPED_DOCUMENTED + ] + assert not stray, f"documented names missing from mortie.__all__: {stray}" + + documented_pairs = { + (module, name) + for module, (_, members) in pages.items() + for name in members + } + stale = MODULE_SCOPED_DOCUMENTED - documented_pairs + assert not stale, f"MODULE_SCOPED_DOCUMENTED entries no longer on a page: {sorted(stale)}" + unreachable = { + (module, name) + for module, name in MODULE_SCOPED_DOCUMENTED + if module.removeprefix("mortie.") not in public + } + assert not unreachable, ( + "module-scoped names must hang off a submodule that is itself in " + f"__all__: {sorted(unreachable)}" + ) From 9d666596b741d10b60518e9ff5c6477affde1480 Mon Sep 17 00:00:00 2001 From: espg Date: Mon, 24 Aug 2026 16:11:27 -0700 Subject: [PATCH 3/6] fold review: state the real no-render reason for the arrow classes (issue #176) --- docs/arrow_interchange.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/arrow_interchange.md b/docs/arrow_interchange.md index 44459e54..94bfe7c4 100644 --- a/docs/arrow_interchange.md +++ b/docs/arrow_interchange.md @@ -20,13 +20,19 @@ pyarrow. ## The pyarrow extension classes: `MortonIndexType` / `MortonIndexExtArray` The pyarrow skin's two classes are public as `mortie.MortonIndexType` and -`mortie.MortonIndexExtArray` (and on `mortie.arrow`), but they are **built -lazily on first attribute access** behind a module `__getattr__`, so that -importing mortie never imports pyarrow. That is also why they have no -rendered [API page](api/arrow.md): mkdocstrings resolves modules statically, -and the classes do not exist until first touched. They are documented here -instead. Touching either name without pyarrow installed raises an -`ImportError` pointing at the missing extra. +`mortie.MortonIndexExtArray` (and on `mortie.arrow`), but they are **defined +inside `_build_type()`** and handed out by a module `__getattr__` — they are +never bound as module attributes. That is why they have no rendered +[API page](api/arrow.md): mkdocstrings resolves modules statically, and +static resolution finds no such attribute to render, whether or not pyarrow +is installed. They are documented here instead. + +pyarrow itself stays **optional**: importing mortie never *requires* it — a +numpy-only install imports fine, and touching either name there raises an +`ImportError` pointing at the missing extra. When pyarrow *is* installed, +`mortie.arrow` builds and registers the extension type eagerly at import, so +a parquet read resolves the `mortie.morton_index` extension name without the +user having touched the type first. - **`MortonIndexType`** is the `pyarrow.ExtensionType` subclass over `uint64` storage with extension name `mortie.morton_index`. It carries no From f511cb6a4e078813a93b86e67085575c40cbf168 Mon Sep 17 00:00:00 2001 From: espg Date: Mon, 24 Aug 2026 16:12:06 -0700 Subject: [PATCH 4/6] fold review: to_numpy widens to float64 once nulls are present (issue #176) --- docs/arrow_interchange.md | 11 +++++++---- mortie/arrow.py | 9 +++++++-- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/arrow_interchange.md b/docs/arrow_interchange.md index 94bfe7c4..d7ead389 100644 --- a/docs/arrow_interchange.md +++ b/docs/arrow_interchange.md @@ -43,10 +43,13 @@ user having touched the type first. - **`MortonIndexExtArray`** is the matching `pyarrow.ExtensionArray` subclass: what `from_morton_index` returns, and what pyarrow hands back when the registered type resolves on read. Its one addition over the - stock class is `to_numpy(**kwargs)`, which materializes the `uint64` - storage (defaulting `zero_copy_only=False` so a null-bearing array - converts); for the null → sentinel-`0` word mapping, go through - `to_morton_index` instead. + stock class is `to_numpy(**kwargs)`, which materializes the storage with + `zero_copy_only=False` by default. That default only stops a null-bearing + array from *raising*: null-free storage comes back as the `uint64` words, + but any null present makes pyarrow widen the result to `float64` with + `NaN` — lossy for 64-bit words. Whenever nulls are possible, go through + `to_morton_index` instead: it fills nulls with the sentinel-`0` word and + keeps `uint64`. ## Producing a column (any Arrow lib) diff --git a/mortie/arrow.py b/mortie/arrow.py index 85466076..4d6059e5 100644 --- a/mortie/arrow.py +++ b/mortie/arrow.py @@ -96,12 +96,17 @@ def to_numpy(self, **kwargs): ---------- **kwargs Forwarded to ``pyarrow.Array.to_numpy``; ``zero_copy_only`` - defaults to ``False`` so a null-bearing array converts. + defaults to ``False`` so a null-bearing array converts + instead of raising. Returns ------- numpy.ndarray - The ``uint64`` packed words. + The ``uint64`` packed words when the storage has no nulls. + With nulls present pyarrow widens the result to ``float64`` + with ``NaN`` in their place, which is lossy for 64-bit + words — use ``to_morton_index`` instead, which fills nulls + with the sentinel ``0`` word and stays ``uint64``. """ kwargs.setdefault("zero_copy_only", False) return self.storage.to_numpy(**kwargs) From e9af4c4a3832e07d78e8ece22227ae1fc0d06687 Mon Sep 17 00:00:00 2001 From: espg Date: Mon, 24 Aug 2026 16:12:50 -0700 Subject: [PATCH 5/6] fold review: fail loud when two pages document one module (issue #176) --- mortie/tests/test_docs_api_pages.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/mortie/tests/test_docs_api_pages.py b/mortie/tests/test_docs_api_pages.py index 1d72e8d8..6db7be8a 100644 --- a/mortie/tests/test_docs_api_pages.py +++ b/mortie/tests/test_docs_api_pages.py @@ -60,11 +60,18 @@ def all_pages(): """Map each documented module to its page name and ``members:`` roster.""" - return { - module: (page.name, members) - for page in sorted(API_DIR.glob("*.md")) - for module, members in [page_members(page)] - } + pages = {} + for page in sorted(API_DIR.glob("*.md")): + module, members = page_members(page) + # One page per module is the repo convention, and this dict assumes + # it: a second page on the same module would silently drop the + # earlier page's roster, disarming both reverse directions. Fail + # loud, as page_members does for two blocks on one page. + assert module not in pages, ( + f"{page.name}: {module} is already documented by {pages[module][0]}" + ) + pages[module] = (page.name, members) + return pages def page_members(path): From 86d92bae2ddae9d316d5a56e8758d7792589cf91 Mon Sep 17 00:00:00 2001 From: espg Date: Mon, 24 Aug 2026 16:13:42 -0700 Subject: [PATCH 6/6] =?UTF-8?q?fold=20review:=20point=20convert.md=20at=20?= =?UTF-8?q?the=20real=20=C2=A79=20anchor=20(issue=20#176)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/api/convert.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api/convert.md b/docs/api/convert.md index 67a6a3a4..ae1f2f0a 100644 --- a/docs/api/convert.md +++ b/docs/api/convert.md @@ -12,7 +12,7 @@ Its default, `"authalic"`, maps WGS84 geodetic latitude to authalic latitude on the way into the spherical kernel and back on the way out, so cells are equal-area on the ellipsoid; `latitude="geodetic-spherical"` is the pre-0.10 escape. The two conventions are non-corresponding partitions — see -[specification.md §9](../specification.md#latitude-convention). The +[specification.md §9](../specification.md#9-latitude-convention-authalic-on-wgs84). The `geodetic_to_authalic` / `authalic_to_geodetic` pair below exposes that latitude→latitude mapping on its own.