notebooks: binder-runnable reader demos (hhdc_viewer, waveform_viewer)

.binder/postBuild

@@ -30,3 +30,13 @@ else
     HATCH_VCS_PRETEND_VERSION="0.0.0+binder" \
         python -m pip install --no-cache-dir ".[analysis,catalog,viz]"
 fi
+
+# The reader-only demos (notebooks/hhdc_viewer.ipynb, notebooks/waveform_viewer.ipynb)
+# run on mortie + moczarr and never import zagg internals, so their runtime is NOT a
+# zagg extra -- each notebook carries its own `%pip install` line and stays runnable
+# outside Binder. Two of those packages must be in the IMAGE rather than installed
+# from a cell, because they ship Jupyter frontend extensions that a running kernel
+# cannot pick up: ipympl (the `%matplotlib widget` backend hhdc_viewer's rotatable
+# 3-D view needs) and ipywidgets (the sliders and dropdowns both use). moczarr itself
+# is pure Python, so the notebooks' own `%pip install` covers it either way.
+python -m pip install --no-cache-dir ipympl ipywidgets

README.md

@@ -146,9 +146,13 @@ The notebooks under `notebooks/` run on [Binder](https://mybinder.org/v2/gh/engl
 | `jupyterhub_example.ipynb` | Drive the API from a science hub; read & visualize a published result | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/jupyterhub_example.ipynb) |
 | `cryocloud_example.ipynb` | End-to-end ISMIP6 read + **AWS Lambda fan-out** on CryoCloud | **not Binder-runnable** (needs live AWS + Earthdata credentials) |
 | `cost_reporting.ipynb` | Max / estimated / actual invoke cost + progress-bar dispatch wrapper | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/cost_reporting.ipynb) |
+| `hhdc_viewer.ipynb` | Polygon → MOC → shard → **rotatable paired 3-D view** (ATL03 + GEDI) → numpy tensors | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/hhdc_viewer.ipynb) |
+| `waveform_viewer.ipynb` | The cell-level join: one GEDI footprint against the 2×2 ATL03 cells beneath it, both from stored digests | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/waveform_viewer.ipynb) |
 
 `cryocloud_example.ipynb` is the only Lambda demo; it dispatches to a deployed AWS Lambda and reads private-account S3 via the CryoCloud IRSA role, so it cannot run on Binder.
 
+`hhdc_viewer.ipynb` and `waveform_viewer.ipynb` are **reader-only**: they run on `mortie` + `moczarr[zagg]` with no zagg-internal imports, reading the anonymous public [source.coop](https://source.coop/englacial/zagg) demo stores (ICESat-2 ATL03 + GEDI L1B over California and the NEON AOP sites). Each carries its own `%pip install` line, so they run outside Binder unchanged. They are split in two because `hhdc_viewer` needs `%matplotlib widget` for its rotatable 3-D view and `waveform_viewer` needs `%matplotlib inline` — the two backends collide in one kernel.
+
 ## Project Structure
 
 ```

notebooks/hhdc_viewer.ipynb

@@ -0,0 +1,239 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "039315dd",
+   "metadata": {},
+   "source": [
+    "# polygon \u2192 MOC \u2192 shard \u2192 3-D \u2192 numpy\n",
+    "\n",
+    "The whole zagg read stack in five functions, two `pip install`s, and zero\n",
+    "credentials. A geojson polygon becomes a morton MOC; the MOC checks itself\n",
+    "against the store's own coverage; the covered shards open with timings; one\n",
+    "shard renders in 3-D (ATL03 + GEDI together); the current view exports to a\n",
+    "numpy tensor and saves to disk. Everything below is reader-side \u2014 `mortie`\n",
+    "for the geometry, `moczarr` for the store \u2014 running anonymously against\n",
+    "public S3, binder-ready."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f67cef5b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -q mortie \"moczarr[zagg]>=0.7\" matplotlib ipympl ipywidgets\n",
+    "%matplotlib widget\n",
+    "\n",
+    "import resource\n",
+    "import time\n",
+    "\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "import moczarr as mz\n",
+    "from mortie import moc\n",
+    "\n",
+    "# The drawing lives in viewers.py beside this notebook, so the cells below stay\n",
+    "# about the READ path. Both demo notebooks share it.\n",
+    "from viewers import BLOCK_ORDER, view3d\n",
+    "\n",
+    "# One store per product, each appendable. Coverage answers which ground the\n",
+    "# store holds; the store name never does.\n",
+    "STORES = {\n",
+    "    \"atl03\": (\"s3://us-west-2.opendata.source.coop/englacial/zagg/demo/atl03_tdigest_o9.zarr\", \"19/h_tdigest_signal\"),\n",
+    "    \"gedi\": (\"s3://us-west-2.opendata.source.coop/englacial/zagg/demo/gedi_flux_o9.zarr\", \"18/rx_flux\"),\n",
+    "}\n",
+    "S3 = {\"region\": \"us-west-2\", \"anonymous\": True}\n",
+    "\n",
+    "rss = lambda: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 2**20  # noqa: E731"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4cafa82b",
+   "metadata": {},
+   "source": [
+    "## One polygon in, covered shards out"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "933b33b2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "aoi = {\"features\": [{\"geometry\": {\"coordinates\": [[  # a ~4 km box on the SERC tract\n",
+    "    [-76.56, 38.87], [-76.50, 38.87], [-76.50, 38.91], [-76.56, 38.91], [-76.56, 38.87]\n",
+    "]]}}]}\n",
+    "\n",
+    "q = moc(aoi)\n",
+    "shards = None\n",
+    "for name, (root, _field) in STORES.items():\n",
+    "    assert mz.coverage_moc(root, **S3).contains(q), f\"{name} does not contain the polygon\"\n",
+    "    ids = set(mz.candidate_shards(root, aoi=q, **S3))\n",
+    "    shards = ids if shards is None else shards & ids\n",
+    "shards = sorted(shards)\n",
+    "shards"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1634ec9a",
+   "metadata": {},
+   "source": [
+    "## Open one shard \u2014 every dataset, timed"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0300cfdb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def open_shard(shard):\n",
+    "    \"\"\"Open the shard's leaf in every store; print open+sweep timing and RSS.\"\"\"\n",
+    "    handles = {}\n",
+    "    for name, (root, field) in STORES.items():\n",
+    "        t0, r0 = time.perf_counter(), rss()\n",
+    "        store = mz.open_leaf(root, shard, **S3)\n",
+    "        _, element = mz.open_ragged(store, field)\n",
+    "        n = sum(len(v) for _, v in mz.read_ragged(store, field))\n",
+    "        print(f\"{name:6s} open+sweep {time.perf_counter() - t0:5.1f}s \"\n",
+    "              f\"(+{rss() - r0:4.0f} MB) \u2014 {n:,} centroids, element {element}\")\n",
+    "        handles[name] = (store, field)\n",
+    "    return handles\n",
+    "\n",
+    "\n",
+    "handles = open_shard(shards[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "138d32bc",
+   "metadata": {},
+   "source": [
+    "## The 3-D view \u2014 both sensors, exact centroids, time-aware"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0e1b012c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "view = view3d(handles, shards[0])\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e70211aa",
+   "metadata": {},
+   "source": [
+    "## Export what you see \u2014 a numpy tensor, shaped your way"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "68119f92",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def export(view, sensor, n_bins=64, resolution=1.0, fit=\"degrade_resolution\"):\n",
+    "    \"\"\"The block on screen -> (rows, cols, n_bins) numpy tensor + z metadata.\n",
+    "\n",
+    "    `n_bins`/`resolution` set the vertical shape; the xy shape follows the\n",
+    "    sensor's cell order within the o12 block -- 2**(cell_order - 12) a side,\n",
+    "    so 128 for ATL03's o19 cells and 64 for GEDI's o18.\n",
+    "    \"\"\"\n",
+    "    store, field = handles[sensor]\n",
+    "    t, mask, (z0, dz), w = next(\n",
+    "        b for b in mz.read_tensors(store, field, n_bins=n_bins, resolution=resolution,\n",
+    "                                   block_order=BLOCK_ORDER, fit=fit)\n",
+    "        if int(b[3]) == int(view.block)\n",
+    "    )\n",
+    "    print(f\"{sensor}: {t.shape} tensor, z = {z0:.1f} m + bin * {dz:g} m\")\n",
+    "    return t, {\"z0\": z0, \"dz\": dz, \"block\": mz.morton_decimal(w)}\n",
+    "\n",
+    "\n",
+    "gedi, meta = export(view, \"gedi\", n_bins=128, resolution=0.5)\n",
+    "np.save(f\"gedi_{meta['block']}.npy\", gedi)\n",
+    "\n",
+    "atl03, meta = export(view, \"atl03\")           # default 64 x 1 m bins\n",
+    "np.save(f\"atl03_{meta['block']}.npy\", atl03)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "585bbd96",
+   "source": [
+    "# ---- second export: every block of the shard as a 128 x 128 x 128 cube ----\n",
+    "# ATL03's cells are o19, so an o12 block is 2**(19-12) = 128 cells a side;\n",
+    "# ask for 128 z-bins at 1 m and the tensor is a cube. One `read_tensors`\n",
+    "# sweep yields one cube per populated block, so a shard exports a SET.\n",
+    "#\n",
+    "# `fit=\"degrade_resolution\"` keeps a block whose trimmed relief exceeds 128 m\n",
+    "# rather than refusing it -- the z gain comes back per block, so the ones that\n",
+    "# did not fit at 1 m say so instead of silently pretending.\n",
+    "cubes = {}\n",
+    "for tensor, mask, (z0, dz), w in mz.read_tensors(\n",
+    "    *handles[\"atl03\"], n_bins=128, resolution=1.0, block_order=BLOCK_ORDER,\n",
+    "    fit=\"degrade_resolution\",\n",
+    "):\n",
+    "    cubes[mz.morton_decimal(w)] = {\"tensor\": tensor, \"z0\": z0, \"dz\": dz,\n",
+    "                                   \"cells\": int(mask.astype(bool).sum())}\n",
+    "\n",
+    "exact = [b for b, c in cubes.items() if c[\"dz\"] == 1.0]\n",
+    "print(f\"{len(cubes)} blocks -> {len(cubes)} cubes of {next(iter(cubes.values()))['tensor'].shape}\")\n",
+    "print(f\"  {len(exact)} at the requested 1 m bins; \"\n",
+    "      f\"{len(cubes) - len(exact)} degraded to fit their relief in 128 bins\")\n",
+    "for b, c in sorted(cubes.items())[:5]:\n",
+    "    print(f\"  {b}  z = {c['z0']:8.1f} m + bin * {c['dz']:.2f} m   \"\n",
+    "          f\"{c['cells']:5,} populated cells   {c['tensor'].nbytes / 2**20:.1f} MiB\")\n",
+    "\n",
+    "np.savez_compressed(\n",
+    "    f\"atl03_cubes_{shards[0]}.npz\",\n",
+    "    **{b: c[\"tensor\"] for b, c in cubes.items()},\n",
+    ")\n",
+    "print(f\"saved atl03_cubes_{shards[0]}.npz\")"
+   ],
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "99e36ee1",
+   "metadata": {},
+   "source": [
+    "Five functions, two libraries, one polygon \u2014 coverage, shards, timings,\n",
+    "the paired 3-D view, and tensors on disk."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}