Skip to content

Publish a before/after report and a release for every data change - #28

Merged
drewvolz merged 31 commits into
mainfrom
change-report
Oct 24, 2026
Merged

drewvolz merged 31 commits into
mainfrom
change-report

Conversation

@drewvolz

@drewvolz drewvolz commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

The college's data changes without anyone looking: the monthly scrape commits straight to main, and its only trace is a bot commit. This builds a page that shows each change, before and after, and publishes a GitHub release linking to it, so watching the repo for releases is how to hear about changes. Data still flows to main unattended; the report explains a change and never holds it back.

The report

uv run scripts/report.py --before REV [--after REV] --date YYYY-MM-DD --out PAGE [--notes NOTES ...] compares two states of the published files (map.json, map.geojson, data/*.geojson) and writes one self-contained HTML page: an overview map framed on the places that changed, old shapes dashed and new solid, dropped and added stretches highlighted, close-ups per area, background-layer changes drawn faintly, and a card per place with fields before and after (prose as a word diff). Same inputs give the same bytes.

  • scripts/changes.py: read a state from git or the working tree; compare place by place (renames are paired from a removed and an added place within 25 m); measure moves. A move is a stretch more than 3 m from the other shape, sampled about every metre against the other shape's outline, so a trail cut in half or a footprint redrawn inside its old outline shows. Smaller redraws are said, never hidden.
  • scripts/report.py: areas, the plain summary, release notes, the command.
  • scripts/report_page.py: the page.

Releases

  • release.yml (new) is the only thing that publishes a release. A person's push that changes the data starts it; the scrape starts it with workflow_dispatch after committing, because a push made with GITHUB_TOKEN starts no workflow. It publishes "Campus data YYYY-MM-DD" (-2 for a second change that day) with the summary and counts, commits the page to site/changes/, and dispatches the tile build so the page goes live. Re-running it for the same change is safe.
  • scrape.yml builds a report on every run, including one that fails verify (uploaded as an artifact then, and on PRs), and dispatches the release after its commit.
  • tiles.yml: the scrape's data commits never triggered a tile build (confirmed on the Sept 30 commit); its comment said they did. The release workflow's dispatch now delivers them within minutes.

Real changes, as the generator renders them

Testing

198 passed, 1 skipped (htmlq absent locally); ruff and actionlint clean. Two tests run the page's hover script in headless Chrome and skip where there is none. A golden page pins the output.

First runs on this PR: release.yml's dry run passed (it found no change to the published data, as expected for a PR that changes none), and CI ran the headless-Chrome tests (199 passed). The scrape dry run fails on the stale Heath Creek FIDs, as on every branch until #27 merges. The real release path (commit, tag, tile dispatch) first runs on the first data change after merge.

Known, not fixed here

  • A close-up caption covering both a moved point and a moved line doesn't say which place moved.
  • A manual dispatch with a different before but the same after reuses that release's name.
  • GitHub keeps one pending run per concurrency group, so a third release queued within minutes drops the pending one.
  • Polygon areas aren't shown on cards; the PR drift caveat is a fixed sentence rather than computed.

Say why field_changes, compare_places and compare_context work the way they do.
… point moved

The summary counts the background shapes that appeared and disappeared per layer, so a change only to those layers no longer reads as blank. The overview draws them faintly in the dropped and added colours and frames itself on them, so such a change gets a map. A moved point says how far it moved instead of 0 m dropped and added.
… or releasing twice

A commit that already has a release keeps its name, the release comes first so a page on main always has one, the page is written outside the checkout and copied onto main's tip, and the commit and push happen only when the page is new.
A background layer re-cut across the whole campus widened the overview
until the changed places shrank into a corner, and a page with one area
has no close-up to fall back on. The places decide the frame; background
shapes are drawn where they reach into it and counted in the notes.
A release that only re-cut a background layer showed a places table of
zeros under a summary naming the layer. The notes now count each layer's
shapes added and removed, and show the places table, like the page's
count tiles, only when a place changed.
Leaving a card cleared the highlight at once, so the gap to the next card
undimmed the map for a few frames and it flickered. Letting go now waits
100 ms, long enough to reach the next card, and stopping on the background
still lets go. Tested by replaying the pointer's events in headless Chrome.
The unnamed ponds, fields, lots and access trails are published only in data/,
so a change to one was in no report and got no release. They are now the
background of their layer, and named ponds are drawn as water from their
place shapes. A layer compares as a multiset, so deleting one of two
identical copies is a change, and a feature with no coordinates is not a
shape. A place is called by its most specific kind, with a word for each
kind in the data.
…ered

A walk's minutes changing rendered as an empty list, because a dict was keyed
by its keys alone; each changed key now shows before and after. A list whose
items only moved, or lost a repeat, also rendered empty: repeats now count,
and a pure reorder says so.
A change with no coordinates reached min() on an empty list, and a crash in
report.py fails the release. Like a text-only change, it has a card and no
place on the map.
A routing-only change gets no report by design, but the run said there was no
change to the published data while committing routing.json.
@drewvolz
drewvolz merged commit 64c22dc into main Oct 24, 2026
7 of 8 checks passed
@drewvolz
drewvolz deleted the change-report branch October 24, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant