Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 4 additions & 8 deletions docs/api/convert.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,9 @@ latitude→latitude mapping on its own.
- mort2healpix
- mort2norm
- norm2mort
- geo2uniq
- norm2uniq
- uniq2geo
- unique2parent
- geodetic_to_authalic
- authalic_to_geodetic

!!! note "Not yet documented here"

The UNIQ helpers (`geo2uniq`, `norm2uniq`, `uniq2geo`, `unique2parent`) are
omitted while their signatures are in flux — see
[issue #136](https://github.com/espg/mortie/issues/136). `heal_norm` is
omitted because it is being removed under
[PR #130](https://github.com/espg/mortie/pull/130).
7 changes: 7 additions & 0 deletions docs/api/coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,14 @@ public entry point, `polygons_to_morton_mocs`, lives in
multipart form is reached through `from_geometry` / `from_wkb` / `from_wkt`
with `moc=True`, or `mortie.Moc`).

The ring-validity checks below report whether any documented winding
convention is in play for a ring *before* covering it — see
[Ring validity](../coverage_methods.md#ring-validity) for the narrative.

::: mortie.coverage
options:
members:
- morton_coverage
- ring_validity
- ring_is_simple
- RingValidity
1 change: 1 addition & 0 deletions docs/api/moc_object.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,4 @@ answer can err near a boundary.
options:
members:
- Moc
- moc
19 changes: 19 additions & 0 deletions docs/api/rank_xy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# mortie.rank_xy

Subtree-local rank <-> face-local `(x, y)` bit deinterleave for 2-D block
views (issue #149). A depth-`d` subtree holds `4**d` cells whose ascending
packed-word order is a Z-order (morton) curve over a `2**d x 2**d` block;
`rank_to_xy` / `xy_to_rank` convert between a cell's **rank** in that block
and the deinterleaved pair, matching the healpy / HEALPix C++ `pix2xyf`
convention (origin at the subtree's south corner). The input is rank-space,
**not** packed morton words — strip the shard prefix down to the base-4
digit-tail rank first. Normative statement:
[specification.md §8](../specification.md#8-rank-space-x-y-deinterleave);

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

This anchor (and ../coverage_methods.md#ring-validity on docs/api/coverage.md) resolves correctly today — I built the site and confirmed specification/index.html carries id="8-rank-space-x-y-deinterleave" and coverage_methods/index.html carries id="ring-validity" — but the PR’s “mkdocs build --strict: green” does not cover it: mkdocs 1.6 defaults validation.links.anchors to info (.venv/.../mkdocs/config/defaults.py:194), and mkdocs.yml declares no validation: block, so a wrong fragment builds clean.

That gap is already live in the tree, including in a file this PR edits:

$ grep -rn "#latitude-convention" docs/
docs/healpix_interchange.md:83 / docs/index.md:55 / docs/api/convert.md:15 / docs/specification.md:315,318
$ grep -o "id=\"[^\"]*latitude[^\"]*\"" site/specification/index.html
id="9-latitude-convention-authalic-on-wgs84"

Five links point at an anchor the built page does not have, and --strict has never said so — the same silent-drift shape this PR exists to close, one layer up from members:.

Fix direction: add validation: {links: {anchors: warn}} to mkdocs.yml so --strict actually pins the new cross-page anchors; that turns the five #latitude-convention links red, so either fix them in the same change or leave both for a follow-up issue — your call on scope, since the dangling ones are pre-existing.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Partial fix in 86d92ba, with the config change and the four other dangling links deliberately left standing.

(1) The anchors this PR adds are verified. Built the site and grepped the HTML: specification/index.html carries id="8-rank-space-x-y-deinterleave" and coverage_methods/index.html carries id="ring-validity", so both new cross-page links resolve today — confirming your check independently rather than resting on --strict, which as you show says nothing about fragments (mkdocs 1.6 defaults validation.links.anchors to info and mkdocs.yml declares no validation: block).

(2) Not turning on validation.links.anchors. Changing mkdocs.yml alters what the Docs build enforces repo-wide and would turn the five pre-existing #latitude-convention links red — a behavior change past this PR’s scope (documenting the public surface). That is a scope call for espg; worth its own issue alongside question (3) on the PR body, since both are "make the docs build assert what we verified by hand".

(3) Fixed the one link in a file this PR already edits. docs/api/convert.md:15 now points at ../specification.md#9-latitude-convention-authalic-on-wgs84 — the real rendered id, read off the built spec page (grep -o id="[^"]*latitude[^"]*" → only 9-latitude-convention-authalic-on-wgs84), not guessed. Rebuilt after: mkdocs build --strictgreen and the rendered href is../../specification/#9-latitude-convention-authalic-on-wgs84`.

The other four (docs/healpix_interchange.md:83, docs/index.md:55, docs/specification.md:315,318) are outside this PR’s file set and stay standing for espg — they are pre-existing and fixing them here would be an unrelated docs sweep.

the public functions ship the Rust kernel (`src_rust/src/rank_xy.rs`). The
names stay flat on the package (`mortie.rank_to_xy`, `mortie.xy_to_rank`).

::: mortie.rank_xy
options:
members:
- rank_to_xy
- xy_to_rank
1 change: 1 addition & 0 deletions docs/api/toc_object.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,4 @@ each answer can err near a span edge (the quanta are ~2–4 s).
options:
members:
- Toc
- toc
25 changes: 25 additions & 0 deletions docs/arrow_interchange.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,31 @@ workers (e.g. an AWS Lambda layer without pyarrow). The raw Arrow C structs are
built in Rust (via the `arrow` crate), so nothing on the critical path imports
pyarrow.

## The pyarrow extension classes: `MortonIndexType` / `MortonIndexExtArray`

The pyarrow skin's two classes are public as `mortie.MortonIndexType` and
`mortie.MortonIndexExtArray` (and on `mortie.arrow`), but they are **built
lazily on first attribute access** behind a module `__getattr__`, so that
importing mortie never imports pyarrow. That is also why they have no

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

“so that importing mortie never imports pyarrow” is false whenever pyarrow is installed, so the sentence states the opposite of the contract the code actually keeps. mortie/arrow.py ends with an eager registration block:

try:
    import pyarrow as _pa  # noqa: F401

    _build_type()
except ImportError:
    pass

and mortie/__init__.py does from . import arrow eagerly — so import mortie pulls pyarrow in and builds/registers the extension type at import time:

$ .venv/bin/python -c "import sys, mortie; print(\"pyarrow\" in sys.modules, mortie.arrow._EXT_TYPE is not None)"
True True

The next clause inherits the same problem: “the classes do not exist until first touched” — with pyarrow present the type object exists from import. What is true (and is the real griffe reason) is that the two names are never module-level attributes: they are locals of _build_type(), reachable only through arrow.__getattr__, so static resolution has nothing to render either way.

Fix direction: state the guarantee the code makes — importing mortie never requires pyarrow (a numpy-only install imports fine, and touching either name raises the curated ImportError) — and pin the no-render reason on “never bound as module attributes” rather than on “not built yet”.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Fixed in 9d66659 — the sentence stated the opposite of the contract, exactly as you show.

The subsection now pins the no-render reason on the binding, not on build timing: the classes are defined inside _build_type() and handed out by a module __getattr__, so they are never bound as module attributes and static resolution finds nothing to render whether or not pyarrow is installed. The optionality claim is re-scoped to what the code actually keeps — importing mortie never requires pyarrow (numpy-only install imports fine, touching either name there raises the curated ImportError) — and the eager block you quoted is now stated positively: when pyarrow is present, mortie.arrow builds and registers the extension type at import so a parquet read resolves the extension name without the user touching the type first.

The LAZY_ARROW_UNDOCUMENTED comment in mortie/tests/test_docs_api_pages.py already carried the accurate version of this reason ("defined inside _build_type() and reached only through arrow.__getattr__"), so the prose and the roster now say the same thing.

rendered [API page](api/arrow.md): mkdocstrings resolves modules statically,
and the classes do not exist until first touched. They are documented here
instead. Touching either name without pyarrow installed raises an
`ImportError` pointing at the missing extra.

- **`MortonIndexType`** is the `pyarrow.ExtensionType` subclass over
`uint64` storage with extension name `mortie.morton_index`. It carries no
parameters — its serialized form is empty; the extension name is the whole
identity — so the type survives parquet / IPC round-trips.
`morton_index_type()` builds, registers, and returns the singleton
instance; there is no reason to construct the class directly.
- **`MortonIndexExtArray`** is the matching `pyarrow.ExtensionArray`
subclass: what `from_morton_index` returns, and what pyarrow hands back
when the registered type resolves on read. Its one addition over the
stock class is `to_numpy(**kwargs)`, which materializes the `uint64`
storage (defaulting `zero_copy_only=False` so a null-bearing array

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

“materializes the uint64 storage (defaulting zero_copy_only=False so a null-bearing array converts)” overpromises for exactly the case the parenthetical calls out: a null-bearing array does not convert to packed words, it converts to float64/NaN.

$ .venv/bin/python
>>> a = marrow.from_morton_index(MortonIndexArray(np.array([1, 0, 3], dtype=np.uint64)))
>>> a.null_count, a.to_numpy(), a.to_numpy().dtype
(1, array([ 1., nan,  3.]), dtype("float64"))

(the 0 word is MortonIndexArray._SENTINEL, mortie/pandas.py:140, so it lands as a null in the Arrow storage). zero_copy_only=False only stops it raising; pyarrow still widens an integer array with nulls to float64, which is lossy for 64-bit words. A reader following this bullet gets NaNs where the doc promised uint64.

The to_morton_index steer is correct — it does storage.fill_null(int(MortonIndexArray._SENTINEL)) (mortie/arrow.py:275) — but as written the sentence reads as “both paths give you words, one just maps nulls”.

Fix direction: say to_numpy() returns the uint64 words for a null-free array and float64/NaN once nulls are present, which is why to_morton_index is the path whenever nulls are possible. Worth syncing MortonIndexExtArray.to_numpy’s own docstring (mortie/arrow.py, “Returns … The uint64 packed words”), which has the same gap.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Fixed in f511cb6, and I took the docstring sync you flagged in the same commit (same finding).

The bullet now says the zero_copy_only=False default only stops the call raising: null-free storage comes back as the uint64 words, while any null present makes pyarrow widen the result to float64 with NaN — lossy for 64-bit words — which is why to_morton_index is the path whenever nulls are possible (it fills nulls with the sentinel 0 word and stays uint64).

MortonIndexExtArray.to_numpy’s own Returns section in mortie/arrow.py carried the same gap and now states the null-free / null-bearing split and points at to_morton_index.

Gates on that commit: pytest mortie/tests/test_arrow.py -q 25 passed; numpydoc lint mortie/arrow.py clean; flake8 mortie --select=E9,F63,F7,F82 clean.

converts); for the null → sentinel-`0` word mapping, go through
`to_morton_index` instead.

## Producing a column (any Arrow lib)

`export_c_array` returns the `(schema_capsule, array_capsule)` pair of the
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ nav:
- API reference:
- convert: api/convert.md
- orders: api/orders.md
- rank_xy: api/rank_xy.md
- buffer: api/buffer.md
- coverage: api/coverage.md
- moc kernel: api/moc.md
Expand Down
8 changes: 8 additions & 0 deletions mortie/tests/test_arrow.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,14 @@ def _blocked(name, *args, **kwargs):
raise ImportError("blocked for test")
return real_import(name, *args, **kwargs)

import mortie

# The fresh import below also rebinds the parent package's ``arrow``
# attribute to the transient copy, and ``delitem`` only saves the
# ``sys.modules`` entry -- register the attribute too so ``undo()``
# restores both, keeping ``mortie.arrow`` identical to
# ``sys.modules["mortie.arrow"]`` (the docs pin checks by identity).
monkeypatch.setattr(mortie, "arrow", sys.modules["mortie.arrow"])
monkeypatch.delitem(sys.modules, "mortie.arrow", raising=False)
for mod in list(sys.modules):
if mod == "pyarrow" or mod.startswith("pyarrow."):
Expand Down
137 changes: 131 additions & 6 deletions mortie/tests/test_docs_api_pages.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,64 @@
**stale-entry** direction: every ``members:`` name in ``docs/api/*.md`` must
resolve as an attribute of the module its page documents.

The reverse direction is deliberately not pinned: nothing here requires every
public name to appear on *some* page, so a move that deletes a member from its
old page and forgets to add it to the new one still vanishes silently — as do
the public names that have no ``docs/api`` entry today. Closing that would
mean pinning an explicit roster of undocumented names, a docs-policy decision
rather than a drift check.
The reverse direction is pinned too (issue #176): every non-submodule name in
``mortie.__all__`` must appear on exactly one page — keyed on *(defining
module, name)*, so the arrow skins of core functions stay legal on their own
page — and every submodule in ``__all__`` must have a page whose ``:::`` block
is that module. A move that deletes a member from its old page and forgets
the new one now fails here instead of vanishing silently. The only names
allowed off the pages are the two lazily-built Arrow classes
(``LAZY_ARROW_UNDOCUMENTED``), which mkdocstrings cannot resolve statically.

And the docs cannot outgrow the frozen surface either: every ``members:``
entry must be in ``__all__`` by name, or sit in ``MODULE_SCOPED_DOCUMENTED``
— the documented names deliberately reached through their submodule
(``mortie.arrow.export_c_array``, ``mortie.morton_index.MortonIndexScalar``)
rather than flat. Growing either roster is a deliberate act reviewed here,
not a silent omission.
"""

import importlib
import inspect
import re
from pathlib import Path

import pytest

API_DIR = Path(__file__).resolve().parents[2] / "docs" / "api"

# The two pyarrow extension classes are defined inside ``_build_type()`` and
# reached only through ``arrow.__getattr__`` (pyarrow is an optional extra),
# so griffe's static resolution cannot see them and no ``members:`` entry can
# render them. They are documented narratively in docs/arrow_interchange.md
# instead — the structural reason will not expire, so this roster should
# never grow (issue #176). Never ``getattr`` these here: resolving them
# raises ImportError when pyarrow is absent.
LAZY_ARROW_UNDOCUMENTED = frozenset({"MortonIndexType", "MortonIndexExtArray"})

# Documented names deliberately *not* flat on the package: each is public as
# an attribute of a submodule that is itself in ``__all__``. The C Data
# Interface trio is namespaced interop plumbing (``mortie.arrow.export_c_array``,
# issue #93); ``MortonIndexScalar`` is the repr/scalar type handed back by the
# ExtensionArray, spelled ``mortie.morton_index.MortonIndexScalar`` (#104).
MODULE_SCOPED_DOCUMENTED = frozenset({
("mortie.arrow", "export_c_array"),
("mortie.arrow", "export_c_schema"),
("mortie.arrow", "import_c_array"),
("mortie.morton_index", "MortonIndexScalar"),
})

_MISSING = object()


def all_pages():
"""Map each documented module to its page name and ``members:`` roster."""
return {
module: (page.name, members)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

all_pages() keys the dict on the module, so if two pages ever document the same module the later-sorted page silently replaces the earlier one’s roster — the very mis-association page_members() refuses to tolerate within a page (“Fail loud here rather than mis-associate”) is silently allowed across pages, and it disarms the reverse direction.

Verified by copying docs/api to a scratch dir and repointing API_DIR at it (working tree untouched): added zz_coverage_split.md carrying a second ::: mortie.coverage block with coverage.md’s four members, and appended a non-public - np to coverage.md. All three new tests pass:

M3-two-pages-one-module: test_every_public_name_documented_on_exactly_one_page PASS
M3-two-pages-one-module: test_lazy_arrow_roster_is_not_stale PASS
M3-two-pages-one-module: test_every_documented_name_is_public PASS

whereas the same - np on coverage.md alone fails as designed:

M2-bogus-np: test_every_documented_name_is_public FAIL -> documented names missing from mortie.__all__: ["coverage.md: np"]

Three consequences of the collision: a stray / non-public members: entry on the losing page escapes the docs→__all__ direction entirely; a name genuinely rendered twice escapes the “exactly one page” rule; and when the surviving page does not repeat a member, the forward test reports "<name>: on no page" for a name that is in fact on a page — a misleading failure to debug.

Fix direction: key on the page ({page.name: (module, members)}) and derive the module lookup from the values, or add a loud assert that no module is claimed by two pages, mirroring the one-block-per-page guard just below.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Fixed in e9af4c4 — took the assert option, mirroring the one-block-per-page guard in page_members() rather than re-keying the dict (consumers stay as they are, and the failure names both pages).

all_pages() is now a loop that asserts a module is claimed by at most one page: zz_coverage_split.md: mortie.coverage is already documented by coverage.md.

Re-ran your escape scenario against a scratch copy of docs/api (working tree untouched, no stray files left): with a second ::: mortie.coverage page present, all three tests now fail on that assert instead of passing —

test_every_public_name_documented_on_exactly_one_page: FAIL -> zz_coverage_split.md: mortie.coverage is already documented by coverage.md
test_lazy_arrow_roster_is_not_stale:                   FAIL -> (same)
test_every_documented_name_is_public:                  FAIL -> (same)

Gates: pytest mortie/tests/test_docs_api_pages.py -q -p no:randomly 20 passed; ruff check --select E,F,W,I --ignore E501 and flake8 --select=E9,F63,F7,F82 on the file clean.

for page in sorted(API_DIR.glob("*.md"))
for module, members in [page_members(page)]
}


def page_members(path):
"""The (module, members) a docs/api page declares, from its mkdocstrings block."""
Expand Down Expand Up @@ -55,3 +97,86 @@ def test_every_member_resolves_on_its_module(page):
"(mkdocstrings drops these silently — the rendered page just loses them)"
)
assert len(members) == len(set(members)), f"{page.name}: duplicate members"


def test_every_public_name_documented_on_exactly_one_page():
# The reverse direction of the pin above (issue #176): a name in
# ``mortie.__all__`` that no page lists has no rendered API entry at all
# — deleting ``- morton_buffer`` from buffer.md must fail here, not
# vanish silently. Keyed on (defining module, name) by object identity,
# so a page documenting another module's *skin* of the same name (the
# arrow forms of from_wkb / polygons_to_morton_mocs) neither satisfies
# nor double-counts the flat name.
import mortie

pages = all_pages()
problems = []
for name in mortie.__all__:
if name in LAZY_ARROW_UNDOCUMENTED:
continue # structurally unrenderable; see the roster's comment
obj = getattr(mortie, name)
if inspect.ismodule(obj):
if f"mortie.{name}" not in pages:
problems.append(f"submodule {name}: no docs/api page")
continue
homes = [
fname
for module, (fname, members) in pages.items()
if name in members
and getattr(importlib.import_module(module), name, _MISSING) is obj
]
if len(homes) != 1:
problems.append(f"{name}: on {homes or 'no page'}")
assert not problems, (
"public names must render on exactly one docs/api page "
f"(the page of the module they are bound from): {problems}"
)


def test_lazy_arrow_roster_is_not_stale():
# If someone finds a way to render the lazy classes (stub declarations,
# a griffe extension), the allowlist must shrink in the same change.
import mortie

documented = {name for _, (_, members) in all_pages().items() for name in members}
assert not LAZY_ARROW_UNDOCUMENTED & documented, (
"allowlisted-as-undocumentable names now appear on a page — prune "
f"LAZY_ARROW_UNDOCUMENTED: {sorted(LAZY_ARROW_UNDOCUMENTED & documented)}"
)
missing = LAZY_ARROW_UNDOCUMENTED - set(mortie.__all__)
assert not missing, f"allowlisted names no longer public: {sorted(missing)}"


def test_every_documented_name_is_public():
# The docs cannot outgrow the frozen surface: a ``members:`` entry must
# be reachable from ``mortie.__all__`` — flat by name, or through the
# justified module-scoped roster. Growing MODULE_SCOPED_DOCUMENTED is a
# deliberate, reviewed act (issue #176).
import mortie

pages = all_pages()
public = set(mortie.__all__)
stray = [
f"{fname}: {name}"
for module, (fname, members) in pages.items()
for name in members
if name not in public and (module, name) not in MODULE_SCOPED_DOCUMENTED
]
assert not stray, f"documented names missing from mortie.__all__: {stray}"

documented_pairs = {
(module, name)
for module, (_, members) in pages.items()
for name in members
}
stale = MODULE_SCOPED_DOCUMENTED - documented_pairs
assert not stale, f"MODULE_SCOPED_DOCUMENTED entries no longer on a page: {sorted(stale)}"
unreachable = {
(module, name)
for module, name in MODULE_SCOPED_DOCUMENTED
if module.removeprefix("mortie.") not in public
}
assert not unreachable, (
"module-scoped names must hang off a submodule that is itself in "
f"__all__: {sorted(unreachable)}"
)
Loading