Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
ae00901
notebooks: binder-runnable reader demos (hhdc_viewer, waveform_viewer)
espg Aug 26, 2026
1b96153
notebooks: share the viewers in viewers.py; say 'contain' in the cove…
espg Aug 26, 2026
347e72e
viewers: binned tensor mode in the 3-D view, checked by default
espg Aug 26, 2026
bb797b2
notebooks: archive the stale examples off main; drop leaked .npy arti…
espg Aug 26, 2026
c0f38f8
fold review: ruff check + format clean on both notebooks (issue #328)
espg Aug 27, 2026
a92d84a
fold review: bin the shard's tensors as they arrive, not all at once …
espg Aug 27, 2026
dd59f40
fold review: put binned points at cell centres, not corners (issue #328)
espg Aug 27, 2026
a3bae14
fold review: grid_xy returns (col, row) so east and north are true (i…
espg Aug 27, 2026
ec66ce9
fold review: bound the figure registry in both interactive views (iss…
espg Aug 27, 2026
cf62491
fold review: split ATL03's chunk side from GEDI's gside (issue #328)
espg Aug 27, 2026
20cdf48
fold review: cache the digest reads and make the empty-cell guard rea…
espg Aug 27, 2026
eefbb1b
fold review: normalize both sensors before ranking on the weaker one …
espg Aug 27, 2026
2912736
fold review: export says whether it got the z resolution it asked for…
espg Aug 27, 2026
3f8919b
fold review: stream the cube export instead of holding the shard in R…
espg Aug 27, 2026
21da9fc
fold review: header says one %pip install and names the one zagg impo…
espg Aug 27, 2026
487fa7f
fold review: cross-link the notebook pair, drop the dangling 07_minim…
espg Aug 27, 2026
96f6d25
fold review: ignore the notebook tensor exports (issue #328)
espg Aug 27, 2026
0df2d62
fold review: say 'no zagg public API', not 'nothing from zagg' (issue…
espg Aug 27, 2026
5ab6913
fold review: drop the duplicated reader-only paragraph (issue #328)
espg Aug 27, 2026
62ca510
fold review: pin the archived-notebook links to a commit permalink (i…
espg Aug 27, 2026
ef5497f
fold review: postBuild names the real zagg dependency and the shadowi…
espg Aug 27, 2026
4410762
fold review: let ipywidgets come from viz rather than reinstalling it…
espg Aug 27, 2026
5400b1d
fold review: say why open_shard's sweep is deliberate and no longer d…
espg Aug 27, 2026
eb7ef39
fold review: the block axes are face-local NE/NW, not compass east/no…
espg Aug 28, 2026
d3c26be
viewers: read the 3-D view's tensors per block, not per shard (PR #540)
espg Aug 28, 2026
616fe4d
notebooks: descending block list; split the export cell; read one blo…
espg Aug 28, 2026
b7fd245
notebooks: say what the timing cell is for, and how to swap the polyg…
espg Aug 28, 2026
b9d7631
viewers: sort the merged 2x2 ATL03 digest; the CDF reads means as int…
espg Aug 28, 2026
d217dc8
notebooks: rewrite the coincident-waveform prose; cite the GEDI gain;…
espg Aug 28, 2026
0d08fd0
notebooks: report reads honestly; co-registered voxel exports (PR #540)
espg Aug 28, 2026
d1372ae
notebooks: unstick the nth slider; block list from the tally, sorted …
espg Aug 29, 2026
7141cf4
notebooks: opacity control, joint-cell ordering, shared shard pick, c…
espg Aug 29, 2026
f33fecc
notebooks: trim the added prose; show both z windows instead of quoti…
espg Aug 29, 2026
06b001b
notebooks: move exports to export.py; isotropic 128^3 chips with repo…
espg Aug 29, 2026
7f40f46
notebooks: reframe the export section around the two decisions it for…
espg Aug 29, 2026
cbd89c9
notebooks: scope the range-bin claim to the examples (PR #540)
espg Aug 29, 2026
7ebed91
notebooks: open_shard -> viewer_stats; the read path is handles plus …
espg Aug 29, 2026
e59ce04
notebooks: paired_blocks helper; drop BLOCK_ORDER and the grid consta…
espg Aug 29, 2026
2ff7a37
viewers: shade by weight ramps down from the slider, not up to opaque…
espg Aug 29, 2026
a076d57
fold review: fit_window clipped and mis-binned silently; unparsable m…
espg Aug 29, 2026
b1b3ab1
viewers: revert continuous_update; bound the nth slider to the block'…
espg Aug 29, 2026
603a6db
viewers: read waveform digests a block at a time; the slider does no …
espg Aug 29, 2026
5933e1b
viewers: keep column sums and gains, not block tensors (1.25 GiB -> ~…
espg Aug 29, 2026
2c7fb10
viewers: opacity fires on release; nth needs no suppression now it re…
espg Aug 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions .binder/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@
# there is a single source of truth for the notebook dependencies (pyproject.toml).
#
# Only PyPI/conda-forge resolvable packages appear here: no ``git+`` URLs and no
# spherely fork. The Binder-runnable notebooks read synthetic data, the public
# anonymous source.coop store, or anonymous CMR-STAC granule *metadata*
# (jupyterhub_example, shardmap_viewer) -- all of which use the default HEALPix
# ``mortie`` backend, so the exact-S2 spherely SpatialIndex backend is never on
# their import path.
# spherely fork. The two Binder notebooks (hhdc_viewer, waveform_viewer) READ the
# anonymous public source.coop demo stores and build no catalogs at all, so no
# SpatialIndex backend is on their import path -- exact-S2 spherely least of all.
#
# NOTE (2026-08-26): ``postBuild`` still installs ``[analysis,catalog,viz]``, but
# ``catalog`` (stac-geoparquet) and ``viz`` (ipyleaflet) were carried for the
# jupyterhub_example / shardmap_viewer notebooks, which have been archived off
# main. Neither remaining notebook imports them, so both extras -- and much of
# ``analysis`` -- are now dead weight in the image and could be trimmed to speed
# the build. Left in place rather than trimmed blind: that is a dependency
# change, and the image is not the place to discover a transitive need.
name: zagg-binder
channels:
- conda-forge
Expand Down
10 changes: 10 additions & 0 deletions .binder/postBuild
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,13 @@ else
HATCH_VCS_PRETEND_VERSION="0.0.0+binder" \
python -m pip install --no-cache-dir ".[analysis,catalog,viz]"
fi

# The reader-only demos (notebooks/hhdc_viewer.ipynb, notebooks/waveform_viewer.ipynb)
# run on mortie + moczarr and never import zagg internals, so their runtime is NOT a
Comment thread
espg marked this conversation as resolved.
Outdated
# zagg extra -- each notebook carries its own `%pip install` line and stays runnable
# outside Binder. Two of those packages must be in the IMAGE rather than installed
# from a cell, because they ship Jupyter frontend extensions that a running kernel
# cannot pick up: ipympl (the `%matplotlib widget` backend hhdc_viewer's rotatable
# 3-D view needs) and ipywidgets (the sliders and dropdowns both use). moczarr itself
# is pure Python, so the notebooks' own `%pip install` covers it either way.
python -m pip install --no-cache-dir ipympl ipywidgets
Comment thread
espg marked this conversation as resolved.
Outdated
21 changes: 10 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,27 +127,26 @@ The store path and output grid parameters are defined in the YAML config (`outpu

### Step 4: Visualize Results

The output Zarr is a public DGGS dataset. The included notebook rasterizes HEALPix cells to a polar stereographic grid for fast rendering with `imshow`.
The output Zarr is a public DGGS dataset. The included notebooks read it back anonymously — a paired 3-D view of one shard, and the cell-level GEDI × ATL03 digest join.

```bash
uv run jupyter notebook notebooks/rasterized_zarr.ipynb
uv run jupyter notebook notebooks/hhdc_viewer.ipynb
```

Adjust `GRID_SPACING` in the notebook to control output resolution.

## Example Notebooks

The notebooks under `notebooks/` run on [Binder](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks) — no install, no credentials. They install `zagg[analysis]` via the `.binder/` conda config and read only synthetic in-notebook data or the **anonymous, public** [source.coop](https://source.coop/englacial/zagg/benchmarks) benchmark store.
The notebooks under `notebooks/` run on [Binder](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks) — no install, no credentials. Both are **reader-only**: they import `mortie` + `moczarr[zagg]` and nothing from zagg itself, reading the anonymous, public [source.coop](https://source.coop/englacial/zagg) demo stores (ICESat-2 ATL03 + GEDI L1B over California and the NEON AOP sites). Each carries its own `%pip install` line, so they run outside Binder unchanged.

| Notebook | What it shows | Binder |
|----------|---------------|--------|
| `custom_aggregations.ipynb` | Config-driven aggregation API on synthetic data | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/custom_aggregations.ipynb) |
| `rasterized_zarr.ipynb` | Rasterize the published HEALPix store to an 8 km polar-stereo grid | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/rasterized_zarr.ipynb) |
| `jupyterhub_example.ipynb` | Drive the API from a science hub; read & visualize a published result | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/jupyterhub_example.ipynb) |
| `cryocloud_example.ipynb` | End-to-end ISMIP6 read + **AWS Lambda fan-out** on CryoCloud | **not Binder-runnable** (needs live AWS + Earthdata credentials) |
| `cost_reporting.ipynb` | Max / estimated / actual invoke cost + progress-bar dispatch wrapper | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/cost_reporting.ipynb) |
| `hhdc_viewer.ipynb` | Polygon → MOC → shard → **rotatable paired 3-D view** (ATL03 + GEDI), binned or exact → numpy tensors | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/hhdc_viewer.ipynb) |
| `waveform_viewer.ipynb` | The cell-level join: one GEDI footprint against the 2×2 ATL03 cells beneath it, both from stored digests | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/englacial/zagg/main?urlpath=lab/tree/notebooks/waveform_viewer.ipynb) |

They share `notebooks/viewers.py`, which holds the drawing so the notebooks stay about the read path. The two are split because one needs `%matplotlib widget` for its rotatable 3-D view and the other `%matplotlib inline`; the backends collide in a single kernel.

**Archived notebooks.** The earlier examples (`custom_aggregations`, `rasterized_zarr`, `jupyterhub_example`, `cryocloud_example`, `cost_reporting`, `aoi_mask`, `shardmap_viewer`, `tdigest_reader_example`, `sentinel2_fusion`) were written against older APIs and had drifted out of date — stale documentation being worse than none. They are preserved on the [`claude/archive-notebooks-2026-08-26`](https://github.com/englacial/zagg/tree/claude/archive-notebooks-2026-08-26/notebooks) branch and removed from `main`.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Four permanent doc links now point into a claude/* working branch, which is the one namespace guaranteed to be unstable.

This line plus docs/aoi_mask.md:201, docs/quickstart.md:67, and docs/quickstart.md:205,215 now resolve through claude/archive-notebooks-2026-08-26. The branch does exist right now —

$ git ls-remote --heads origin claude/archive-notebooks-2026-08-26
c56221b4396ce386836d313137c1f383eb1240d8  refs/heads/claude/archive-notebooks-2026-08-26

— but claude/* is exactly the namespace CLAUDE.md §2 reserves for agent working branches and that routine runs create, push to, and (post-merge) clean up. Published documentation on main pointing at it will break silently the first time someone tidies stale agent branches, and nothing in the branch name says "do not delete me".

Two stable alternatives, either fine:

  • A commit permalink. c56221b4 is the last commit where those notebooks lived on main, and GitHub keeps blob URLs at an explicit sha forever, independent of any branch: https://github.com/englacial/zagg/blob/c56221b4/notebooks/aoi_mask.ipynb. Zero new refs to maintain.
  • A tag (e.g. notebooks-archive-2026-08-26) — but note tag creation in this repo is a production trigger (publish.yml fires on *.*.*), so it must not look like a version, and §1 puts tag pushes out of an agent's hands anyway.

Separately: this is a scope change the PR body does not describe. The body opens "Two reader-only demo notebooks … plus the .binder and README wiring they need"; bb797b2 additionally deletes nine notebooks (3,832 lines) from main and rewrites three docs/ pages. Under §2 that belongs in the description with its own phase entry — and under §6 a scope change of this size is an @espg decision rather than something a review round should quietly absorb.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Split this one: the link half is folded, the scope half is not mine to take and stays open.

Folded (62ca510) — your first alternative, the commit permalink. All four links now resolve through c56221b4 instead of claude/archive-notebooks-2026-08-26:

  • README.md:147, docs/aoi_mask.md:201, docs/quickstart.md:67, docs/quickstart.md:205 and :215.

Verified c56221b4 actually carries them before repointing — git ls-tree c56221b4 notebooks/ lists all nine (aoi_mask, cost_reporting, cryocloud_example, custom_aggregations, jupyterhub_example, rasterized_zarr, sentinel2_fusion, shardmap_viewer, tdigest_reader_example). It is this PR's own base, i.e. the last commit on main that had them. The README prose changed too, since it said "branch" while pointing at a sha:

They are removed from main and preserved at c56221b4, the last commit on main that carried them — a commit permalink rather than a branch link, because GitHub keeps a blob URL at an explicit sha forever and there is no ref anyone has to remember not to delete.

I did not create a tag; agents do not push tags here, and a *.*.* tag triggers publish.yml.

Left standing — the scope point. You are right that bb797b2a deletes nine notebooks and rewrites three docs/ pages while the PR body still opens "two reader-only demo notebooks … plus the .binder and README wiring they need". That is a scope change, and a review round is not what should absorb it. I have described it in the PR body under the phases checklist and raised it as an explicit question rather than acting further on it — reverting the archive, or keeping it and restating the PR's scope, is a call for the maintainer, not for this fold.

Leaving this thread unresolved for that reason.


Generated by Claude Code


`cryocloud_example.ipynb` is the only Lambda demo; it dispatches to a deployed AWS Lambda and reads private-account S3 via the CryoCloud IRSA role, so it cannot run on Binder.
`hhdc_viewer.ipynb` and `waveform_viewer.ipynb` are **reader-only**: they run on `mortie` + `moczarr[zagg]` with no zagg-internal imports, reading the anonymous public [source.coop](https://source.coop/englacial/zagg) demo stores (ICESat-2 ATL03 + GEDI L1B over California and the NEON AOP sites). Each carries its own `%pip install` line, so they run outside Binder unchanged. They are split in two because `hhdc_viewer` needs `%matplotlib widget` for its rotatable 3-D view and `waveform_viewer` needs `%matplotlib inline` — the two backends collide in one kernel.
Comment thread
espg marked this conversation as resolved.
Outdated
Comment thread
espg marked this conversation as resolved.
Outdated

## Project Structure

Expand Down
2 changes: 1 addition & 1 deletion docs/aoi_mask.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ strict = ds.where(ds["aoi_mask"])
equivalent `(lats, lons)` ring.

See the runnable, data-free example in
[`notebooks/aoi_mask.ipynb`](https://github.com/englacial/zagg/blob/main/notebooks/aoi_mask.ipynb),
[`aoi_mask.ipynb`](https://github.com/englacial/zagg/blob/claude/archive-notebooks-2026-08-26/notebooks/aoi_mask.ipynb) (archived — see the note below),
which builds a small HEALPix grid + AOI box and shows the mask is `True` exactly
for the in-AOI cells. The notebook is self-contained (no remote data) and runs
anywhere `zagg` (with `mortie>=0.8.3`) is installed. Binder launch additionally
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ See [Lambda Deployment](deployment/lambda.md) for AWS setup.
### 3. Visualize results

```bash
uv run jupyter notebook notebooks/rasterized_zarr.ipynb
uv run jupyter notebook notebooks/hhdc_viewer.ipynb
```

## Design Philosophy
Expand Down
6 changes: 3 additions & 3 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ cells. The processing step consumes this file.
To inspect the chunking interactively -- shard outlines, granule footprints,
and a grid that appears on zoom -- use the shard-map viewer
(`pip install zagg[viz]`). See the
[shard-map viewer notebook](https://github.com/englacial/zagg/blob/main/notebooks/shardmap_viewer.ipynb),
[shard-map viewer notebook](https://github.com/englacial/zagg/blob/claude/archive-notebooks-2026-08-26/notebooks/shardmap_viewer.ipynb) (archived),
which runs on a synthetic example (no network needed) and includes manual
in-browser verification instructions.

Expand Down Expand Up @@ -202,7 +202,7 @@ excludes is reported as **not applicable** — `touch_skipped_paths` in the run
record — never as a failure.

See `src/zagg/configs/atl06.yaml` for a complete example and the
[custom aggregations notebook](https://github.com/englacial/zagg/blob/main/notebooks/custom_aggregations.ipynb)
[custom aggregations notebook](https://github.com/englacial/zagg/blob/claude/archive-notebooks-2026-08-26/notebooks/custom_aggregations.ipynb) (archived)
for customization examples.

## Reading the output
Expand All @@ -212,6 +212,6 @@ reconstructs dense, spatially faithful `(64, 64, n_bins)` tensors — with an
occupancy mask and a shared `(offset, gain)` z-window per block (the
[reader contract](ragged_layout.md#spatially-faithful-tensors-deinterleave-blocks-mask))
— or lossless raw value vectors from the stored digests. The
[t-digest reader notebook](https://github.com/englacial/zagg/blob/main/notebooks/tdigest_reader_example.ipynb)
[t-digest reader notebook](https://github.com/englacial/zagg/blob/claude/archive-notebooks-2026-08-26/notebooks/tdigest_reader_example.ipynb) (archived)
walks through `read_tensors` and `read_raw_values` end to end on a self-contained
synthetic store.
146 changes: 0 additions & 146 deletions notebooks/aoi_mask.ipynb

This file was deleted.

Loading
Loading