From 96b9888caf0b19257127110982c8bc56b8e08d82 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 09:22:25 +0000 Subject: [PATCH 1/7] Add Mishnah Entities knowledge base spec adapted to repo conventions --- README.md | 1 + docs/entities-knowledge-base.md | 410 ++++++++++++++++++++++++++++++++ 2 files changed, 411 insertions(+) create mode 100644 docs/entities-knowledge-base.md diff --git a/README.md b/README.md index 692c680..233a42b 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ This repo contains: - [HTML format spec](docs/html-format.md) — HTML structure, anchors, CSS conventions - [How verification works](docs/how-verify-works.md) — normalization pipeline and comparison logic - [Verification report](docs/verification-report.md) — error categories found and lessons learned + - [Entities knowledge base](docs/entities-knowledge-base.md) — design spec for an accumulating store of people, places, and plants in the Mishnah (v0, foundational) - **Scripts** — full pipeline for downloading, formatting, verifying, fixing, and merging: - `scripts/download.py` — fetch raw JSON from Sefaria API - `scripts/format.py` — format JSON using Ollama, Anthropic API, or Claude Code diff --git a/docs/entities-knowledge-base.md b/docs/entities-knowledge-base.md new file mode 100644 index 0000000..88e28ab --- /dev/null +++ b/docs/entities-knowledge-base.md @@ -0,0 +1,410 @@ +# Mishnah Entities Knowledge Base — Database Spec + +**Status:** v0 (foundational). This spec defines *the database* — the accumulating store of people, places, and plants found in the Mishnah. It does **not** cover the detection pipeline; the detector is a later, separate writer into this store (see §10). + +This is a companion subsystem to the text-formatting pipeline that produces `masechot/`. It is adapted to this repo's existing conventions: the slug rules already used for masechet filenames, the deep-link anchor scheme in [`docs/html-format.md`](html-format.md), the practice of gitignoring derived artifacts, and a flat `scripts/` directory. Where this spec departs from a generic "fresh repo" layout, it is to match those conventions; the departures are called out inline. + +----- + +## 1. What this is, and what it is not + +This is a **knowledge base that accumulates**, not a stateless pipeline. Records are long-lived, enriched from many sources over time, and repeatedly corrected by hand. Design priorities follow from that — not throughput (the corpus is tiny: a few hundred people, a few hundred places, ~150–300 plant terms), but: + +1. **Stable identity** — an entity keeps its key across re-imports and re-identifications. +2. **Provenance** — every fact knows where it came from and when. +3. **Idempotent, mergeable updates** — re-running an import causes no churn and no duplicates. +4. **Protected curation** — automated refreshes never clobber human-confirmed facts. +5. **Native support for contested, multi-valued facts** — especially plant identifications and disputed place coordinates. + +The first goal is this store, standing alone. Mishnah integration comes later. + +----- + +## 2. Architecture: git-tracked YAML as source of truth, derived SQLite as index + +- **Source of truth:** one human-readable **YAML file per entity**, committed to git. +- **Query index:** a **SQLite** database compiled from the YAML by a deterministic build step. +- The SQLite is **disposable and regenerable**. Never hand-edit it. Rebuild from YAML. + +Why this shape fits an accumulating, human-in-the-loop KB: + +- **Git history is the audit log.** Each discovery is a commit; `git blame` on a field shows when and why it changed — provenance-over-time for free. +- **Diffs are reviewable and merge-friendly.** A discovery touches one file; clean diffs, minimal conflicts; an agent can make targeted edits without rewriting a monolith. +- **Hand- and agent-editable**, with adjudication notes living next to the data. +- **You still get SQL** for the eventual map and cross-entity queries by rebuilding the index. + +This is the same shape as the text pipeline already in this repo: a deterministic build from a public-domain source, with the database a *derived artifact*. There, `download.py → format.py → … → merge.py` turns Sefaria source text into `masechot/*.html`, and `verify.py` emits a derived HTML report. Here, `data/**/*.yaml` compiles into `knowledge.db`. In both cases the committed source is authoritative and the built artifact is regenerable (see §11 for how `knowledge.db` is gitignored, just like `sefaria/` and `output/`). + +----- + +## 3. Repository layout + +The KB lives in its own top-level `entities/` directory — paralleling `masechot/`, which holds everything for the text subsystem — so the two subsystems stay cleanly separated. Its build and validation scripts live in the shared `scripts/` directory, alongside the existing pipeline scripts, so all tooling is discoverable in one place (as documented in the README). + +``` +entities/ + data/ + people/ akiva.yaml yehuda-b-ilai.yaml … # one file per entity, filename == slug + places/ tzippori.yaml yavne.yaml … + plants/ chitah.yaml ezov.yaml … + sources.yaml # source registry (§6) + vocab/ # controlled vocabularies + generations.yaml sedarim.yaml halachic-categories.yaml regions.yaml + schema/ # JSON Schema — the contract (§7, §9) + claim.schema.json person.schema.json place.schema.json plant.schema.json source.schema.json + knowledge.db # derived; rebuildable. Gitignored (§11), like sefaria/ and output/. + +scripts/ + kb-build.py # compile entities/data/ -> entities/knowledge.db + kb-validate.py # validate every file against its schema + kb-detect.py # the detector (§10) — added later; one writer among several +``` + +Departures from a generic layout, and why: + +- **`entities/` rather than a root `data/`** — keeps the KB self-contained and avoids ambiguity next to the text pipeline's working dirs (`sefaria/`, `output/`). +- **Build/validate scripts in `scripts/`, not a separate `build/`** — this repo keeps all tooling in a single flat `scripts/` directory (`download.py`, `format.py`, `verify.py`, `fix.py`, `merge.py`, `update-readme.py`). KB scripts follow suit, hyphen-prefixed `kb-` to group them. +- **A *scoped* `entities/CLAUDE.md`, not a root one** — durable invariants for coding agents belong in a `CLAUDE.md` (Claude Code reads nested `CLAUDE.md` files for the subtree they sit in), but scoped to `entities/` so they govern the KB without overriding the text-formatting repo at large. Its canonical text is the appendix at the end of this document, to be lifted into `entities/CLAUDE.md` when the subsystem is scaffolded. See §12 and the appendix. + +----- + +## 4. Identity: mint local IDs; external IDs are attributes + +The **primary key is a locally-minted slug**, which is also the filename: + +- `person:akiva`, `place:tzippori`, `plant:chitah` (the `type:` prefix is implied by the folder; the slug stored in-file is the bare `akiva`). + +This is mandatory here because external authorities don't cover everything — especially plants, which have **no gazetteer**. You need a key that is stable *before* a Wikidata/Pleiades link is resolved, and that **survives an identification changing**. External identifiers are joinable *attributes*, never the primary key: + +- people: `wikidata_qid`, `sefaria_slug`, `hyman_ref`, `bonayich_id` +- places: `wikidata_qid`, `pleiades_id`, `geonames_id`, `sefaria_slug` +- plants: `wikidata_qid`, `gbif_id`, `powo_id` (these live on each identification *candidate*, not the term) + +**Slug rules** — identical to the convention already used for masechet filenames (`bava-kamma`, `rosh-hashanah`, `maaser-sheni`, `avodah-zarah`): lowercase ASCII, hyphen-separated, transliterated; disambiguate collisions with a patronymic or epithet (`yehuda-b-ilai` vs `yehuda-ha-nasi`). Slugs are permanent once minted — renames require an explicit alias, never a silent change. Masechet slugs referenced from entities (e.g. in `appearances`, §7) **must** match the existing `masechot/*.html` filenames exactly, so the two subsystems join cleanly. + +----- + +## 5. The claim model: contested facts are claims, not values + +Any fact that can be disputed, sourced, or revised is stored as a **claim**, not a bare value. One shared shape, used by plant identification candidates, place coordinates, people identity resolution, biographical facts, and more: + +```yaml +- value: # scalar or object + source: # provenance (required) + confidence: accepted | probable | minority | disputed | unknown + asserted_by: # e.g. Feliks, Löw + date: + confirmed: false # true == human-ratified; protected (§8) + note: # adjudication reasoning +``` + +Rules: + +- A field that holds claims holds a **list** of them. One may be marked as the **display/primary** value (`display_taxon` for plants, the chosen coordinate for places); the rest are retained dissent. +- A re-import **appends** a claim; it never overwrites an existing one. +- Simple, uncontestable facts (a Hebrew lemma, a tractate name) may be stored as plain scalars — don't over-model. Use claims where provenance or dispute is real. + +**Status lifecycle** (per entity, top-level `status:`): `stub → enriched → reviewed → confirmed`. + +- `stub` — minted, minimal (often just a name + one appearance). +- `enriched` — auto-populated from imports, unreviewed. +- `reviewed` — a human has looked at it. +- `confirmed` — human-ratified; the strongest protection applies. + +----- + +## 6. Source registry + +Every source is a keyed entry in `entities/data/sources.yaml`. Provenance fields reference the **key** only. + +```yaml +sefaria: + citation: "Sefaria, sefaria.org" + type: structured + license: public-domain # the Mishnah text itself is PD; same source the text pipeline uses + trust_tier: 1 # canonical reference/appearance source for this project +wikidata: + citation: "Wikidata, wikidata.org" + type: structured + license: CC0 + trust_tier: 2 +pleiades: + citation: "Pleiades: A Gazetteer of Past Places, pleiades.stoa.org" + type: structured + license: CC-BY + trust_tier: 1 +low: + citation: "Immanuel Löw, Die Flora der Juden, 4 vols., Vienna 1924–1934" + type: scholarly-prose + license: public-domain # author d. 1944; pre-1929 vols PD, life+70 elapsed + trust_tier: 1 +feliks: + citation: "Yehuda Feliks, works on flora of rabbinic literature" + type: scholarly-prose + license: in-copyright # reference/adjudicate only — do NOT bulk-redistribute + trust_tier: 1 +detector: + citation: "Mishnah entity detector (this project)" + type: derived + license: project + trust_tier: 3 +manual: + citation: "Manual curation" + type: human + license: project + trust_tier: 0 # most trusted; lower number == higher trust +``` + +`sefaria` is a first-class source here because it is the canonical text source for the whole repo — the same source `download.py` fetches and `verify.py` checks against — and supplies the concordance behind `appearances.other` (§7). + +Tracking `license` is not pedantry: it determines what may be redistributed if the dataset is ever published. The repo's posture is that **the underlying Mishnah text is public domain** while project-authored material is MIT (see the README); the KB extends that posture per-source. Löw (public domain) and Feliks (in copyright) differ sharply — Feliks is for adjudication and citation, not bulk extraction. + +----- + +## 7. Shared conventions + +- **Appearances** are stored per entity as a deduped list of refs. The Mishnah refs use **this repo's canonical masechet slugs** (the `masechot/*.html` filenames), so each appearance resolves deterministically to a deep link via the anchor scheme in [`docs/html-format.md`](html-format.md): + + ```yaml + appearances: + mishnah: ["shekalim 4:2", "parah 11:7"] # {masechet-slug} {chapter}:{mishna}; from the detector (later) + other: ["Tosefta Shekalim 2:1"] # from the Sefaria concordance; free-form + ``` + + A `mishnah` ref of the form `{masechet-slug} {N}:{M}` maps to `masechot/{masechet-slug}.html#mishna-{N}-{M}` — e.g. `shekalim 4:2` → `masechot/shekalim.html#mishna-4-2`. Dedupe by normalized ref string. Adding an appearance is an idempotent upsert. (Refs into corpora this repo doesn't format — Tosefta, Bavli — stay free-form under `other:`.) +- **Computed fields are never stored raw.** `contemporaries` (people) and region rollups are derived at build time, not committed. +- All free text may be Hebrew or English; keep Hebrew in its own keyed subfields (`he`, `en`). + +----- + +## 8. Update & merge rules (the heart of an accumulating KB) + +1. **Additive, idempotent upserts**, keyed on `(entity_id, source, field)`. Re-running the Wikidata import produces no duplicates and no diff. +2. **Appearances** dedupe by ref. **Identification candidates / coordinate candidates** append if the `(value, source)` pair is new, else no-op. +3. **Protected curation — the most important rule.** A field whose claim is `confirmed: true` is **never auto-overwritten**. If an automated refresh produces a conflicting value, it does **not** write — it records the conflict to a review queue (`entities/conflicts.log` or an `unreviewed_conflicts:` block on the entity) for a human to adjudicate. +4. **New entity → new file** (status `stub`). Never inline-create an entity inside another. +5. **Merges** (two slugs found to be one entity): keep the surviving slug, add the other to `aliases:`, move appearances and claims over, leave a tombstone file that redirects. Never delete history. + +----- + +## 9. Entity schemas + +YAML files; validated against JSON Schema in `entities/schema/`. Below are the field models; the `*.schema.json` files are the enforceable contract and are the **first thing to scaffold**. + +### 9.1 Person + +```yaml +slug: akiva +status: enriched +type: tanna # tanna | amora | biblical | named-layperson | collective +aliases: [] +names: + he: "רבי עקיבא" + en: "Rabbi Akiva" + variants: ["רבי עקיבא בן יוסף", "ר' עקיבא"] +era: + generation: 3 # tannaitic generation 1–6 — the reliable time unit + floruit: "early 2nd c. CE" + locale: ["bnei-brak"] # slugs into places/ +relationships: # borrowed (Wikidata/biographical); typed edges + teachers: ["eliezer-b-hyrcanus", "yehoshua-b-hananiah"] + students: ["meir", "yehuda-b-ilai", "shimon-b-yochai"] + family: [] + # disputants / colleagues are DERIVED from the corpus at build time, not stored +bio: + - value: "Leading tanna of the 3rd generation; foundational to the Mishnah." + source: wikidata + confidence: accepted + confirmed: false +appearances: + mishnah: [] + other: [] +ids: + wikidata_qid: Q310357 + sefaria_slug: rabbi-akiva + hyman_ref: null + bonayich_id: null +``` + +### 9.2 Place + +```yaml +slug: tzippori +status: enriched +type: settlement # settlement | region | temple_structure | water_feature | legal_domain +aliases: [] +names: + he: "ציפורי" + en: "Tzippori" + variants: ["צפורי"] + classical: ["Sepphoris", "Diocaesarea"] # Greco-Roman toponym; settlements only, often empty + modern: { he: "ציפורי", ar: "صفورية", en: "Tzippori" } +geo: # null block for temple_structure & legal_domain + coordinates: # the displayed value; candidates retain dissent + - value: { lat: 32.7522, lon: 35.2797 } + source: pleiades + confidence: accepted + confirmed: false + region: galilee + modern_admin: "Israel, Northern District" + consensus: identified # identified | probable | disputed | unknown +media: + photo: "https://commons.wikimedia.org/wiki/..." # Wikidata P18 + map_link: "https://www.openstreetmap.org/?mlat=32.7522&mlon=35.2797" +plan: null # temple_structure only: { position, diagram_ref, source_ref } +appearances: + mishnah: [] + other: [] +ids: + wikidata_qid: Q745966 + pleiades_id: "678378" + geonames_id: null + sefaria_slug: null +``` + +For `type: temple_structure`, `geo` is null and `plan` is populated: + +```yaml +plan: + position: "north of the altar" + source_ref: "middot 3:1" # canonical masechet-slug ref (§7) + diagram_ref: "mikdash-plan#lishkat-hagazit" # internal schematic key, not lat/long +``` + +### 9.3 Plant + +```yaml +slug: chitah +status: enriched +term: + he: "חיטה" + variants: ["חטה", "חטים", "חיטים"] # spelling + plural + inflected + en_common: "wheat" +term_type: species # species | genus_or_group | folk_category | product | plant_part +identification: + consensus: identified # identified | contested | unknown + display_taxon: "Triticum aestivum" + candidates: + - value: { taxon: "Triticum aestivum", rank: species } + ids: { wikidata_qid: Q12100, gbif_id: null, powo_id: null } + source: feliks + asserted_by: Feliks + confidence: accepted + confirmed: false + # a contested term carries >1 candidate, e.g. ezov: + # - value: { taxon: "Origanum syriacum", rank: species } confidence: accepted + # - value: { taxon: "Satureja thymbra", rank: species } confidence: minority +names: # keyed to display_taxon; from botanical DBs + botanical: "Triticum aestivum" + family: "Poaceae" + common_en: ["bread wheat", "wheat"] + modern_he: ["חיטה"] + arabic: ["قمح"] +usage: + ethnobotanical: ["food"] + medicinal: # historical/ethnobotanical, NOT medical guidance; provenance-tagged + - value: "—" + source: duke + provenance: "Dr. Duke's Phytochemical & Ethnobotanical DB" + confidence: probable + halachic: ["kilayim", "maaser-dagan", "challah", "chadash"] # DERIVED from the Mishnah +media: + photo: "https://commons.wikimedia.org/wiki/..." # of display_taxon + illustration: "https://commons.wikimedia.org/wiki/...Köhler..." # public-domain plate +appearances: + mishnah: [] + other: [] +ids: {} # term-level external ids rare; real ids live on candidates +``` + +`folk_category` terms (e.g. `kitnit`) carry a **set of member species** instead of a single `display_taxon`, and are excluded from the photo/map layer. `product` and `plant_part` terms are tagged and kept but likewise excluded from rendering. + +----- + +## 10. How the Mishnah detector slots in later + +Designing this as a **multi-writer, provenance-tracked base** is exactly what makes the later bolt-on painless. When the detector is built (as `scripts/kb-detect.py`), it is just **one writer among several** (Wikidata import, Pleiades import, Löw/Feliks adjudication, manual curation, detector) — much as the text pipeline is a chain of small single-purpose scripts. It writes through the same upsert rules (§8) and does only two things: + +1. emits **appearances** (`source: detector`) onto existing entities, using the canonical `{masechet-slug} {N}:{M}` ref form (§7), and +2. proposes **stub entities** (status `stub`) for unrecognized mentions, for human review. + +It never overwrites identifications or biographical facts. Nothing about the detector needs to exist for the KB to be useful now. A natural input for it is the already-verified text in `masechot/` (or the Sefaria source the repo downloads), so detected refs line up exactly with the existing anchor scheme. + +----- + +## 11. Build & validation + +- **Validation is the contract.** `scripts/kb-validate.py` validates every `entities/data/**/*.yaml` against its `entities/schema/*.schema.json`. Wire it into a pre-commit hook and CI; a file that doesn't validate doesn't land. Writing these schemas **is** the concrete first deliverable. +- **Build** (`scripts/kb-build.py`) compiles YAML → `entities/knowledge.db`. Suggested relational layout (flatten, don't mirror the nesting): + - `entity(slug, type, status, he, en)` + - `claim(entity_slug, field, value_json, source, confidence, confirmed, asserted_date, note)` — provenance-tracked fields land here + - `appearance(entity_slug, ref, work, source)` + - `external_id(entity_slug, authority, id)` + - `source(key, citation, type, license, trust_tier)` + - derive `contemporaries`, dispute/co-occurrence edges, and region rollups **at build time** into their own tables. +- The build is deterministic and idempotent: same YAML in → same DB out. +- **`entities/knowledge.db` is a derived artifact and is gitignored**, consistent with how this repo already treats built/working artifacts (`sefaria/`, `output/`, `__pycache__/` are all in `.gitignore`). Add `entities/knowledge.db` and `entities/conflicts.log` to `.gitignore`; never commit the DB. (This resolves the source spec's open question in favor of the repo's established convention: derived data is rebuilt, not committed.) + +----- + +## 12. First deliverables (in order) + +1. **`entities/schema/claim.schema.json`** + **`source.schema.json`** — the shared spine. +2. **`entities/schema/{person,place,plant}.schema.json`** — the three contracts. +3. **`entities/data/sources.yaml`** + **`entities/data/vocab/*`** — registries and controlled vocabularies. The `sedarim`/masechet vocab reuses the existing `masechot/*.html` slugs verbatim, so appearance refs validate against the canonical list. +4. **`scripts/kb-validate.py`** + pre-commit/CI wiring, and the `.gitignore` entries from §11. +5. A handful of **hand-written exemplar entities** (one per type: `akiva`, `tzippori`, `chitah`) that validate — these double as fixtures and as living documentation, exactly as `exemplar-zevachim-1.html` does for the formatting skill. +6. **`scripts/kb-build.py`** producing `entities/knowledge.db`. +7. Only then: importers (Wikidata → people/places; Pleiades → places; Wikidata/Löw → plant candidates). + +Format decision: **YAML for entity files** (hand-curation comfort, inline notes), **JSON Schema validating it**. Keep one entity per file. + +Durable invariants for coding agents working in this subsystem are stated in the appendix below, whose canonical home is a scoped **`entities/CLAUDE.md`** created alongside deliverable 1 — scoped to the subtree so it governs the KB only, not the text-formatting repo at large. (Should the KB later grow interactive tooling, the same invariants can also seed a `.claude/skills/` skill, as the formatting rules do for the `mishnah` skill.) + +----- + +## Appendix: `entities/CLAUDE.md` — agent invariants + +The canonical text of the scoped `entities/CLAUDE.md`. Create it with deliverable 1 (§12). These invariants are non-negotiable; honor them in every change to the KB subtree. Paths are written relative to `entities/` (e.g. `data/` = `entities/data/`). + +> # CLAUDE.md — Mishnah Entities Knowledge Base +> +> This subsystem is an **accumulating knowledge base** of people, places, and plants in the Mishnah — not a stateless pipeline. Full design in [`docs/entities-knowledge-base.md`](entities-knowledge-base.md). The invariants below are non-negotiable; honor them in every change. +> +> ## Architecture +> +> - **Source of truth = one YAML file per entity, in git.** `knowledge.db` (SQLite) is a **derived, regenerable index** — never hand-edit it; rebuild from `data/`. It is gitignored. +> - Git history is the audit log. One discovery = one focused commit touching as few files as possible. +> +> ## Identity +> +> - **Primary key = a locally-minted slug** (= the filename). Slugs are permanent once minted; renames go through `aliases:`, never silent change. Masechet slugs referenced in `appearances` must match the repo's `masechot/*.html` filenames. +> - External IDs (`wikidata_qid`, `pleiades_id`, `powo_id`, …) are **joinable attributes, never the primary key**. An entity must be keyable before any external link is resolved. +> +> ## Facts & provenance +> +> - **Every contestable fact is a claim, not a bare value**: `{value, source, confidence, asserted_by, date, confirmed, note}`. Lists of claims; one may be the display value, the rest retained dissent. +> - **Every claim cites a `source`** that exists in `data/sources.yaml`. No source → no write. +> - Don't over-model: uncontestable scalars (a Hebrew lemma) stay plain. +> +> ## Update rules (an accumulating KB lives or dies by these) +> +> - Imports are **additive and idempotent**, keyed on `(entity_id, source, field)`. Re-running an import must produce **zero churn**. +> - **NEVER auto-overwrite a `confirmed: true` claim.** On conflict, write to the review queue, not the field. +> - Appearances dedupe by ref; candidates append on new `(value, source)`; new entity → new `stub` file; merges leave a tombstone. +> +> ## Contested by design +> +> - Plant identifications and disputed place coordinates are **multi-valued**. Keep all candidates; mark one for display (`display_taxon` / chosen coordinate). Never collapse dissent to force a single answer. +> +> ## Licensing +> +> - Respect `license` in the source registry. **Löw = public domain (extractable); Feliks = in-copyright (adjudicate/cite only, do not bulk-redistribute).** The underlying Mishnah text is public domain; project-authored material is MIT (see the repo README). +> +> ## Validation +> +> - Every `data/**/*.yaml` must validate against its `schema/*.schema.json`. A file that doesn't validate doesn't land (pre-commit + CI). Run `scripts/kb-validate.py`. +> +> ## The detector comes later +> +> - The Mishnah entity detector (`scripts/kb-detect.py`) is **one writer among several**. It only emits appearances and proposes `stub` entities. It must not overwrite identifications or biographical facts. From 61dcfb1172d83c4582ad506c9ff2f23ec4cb3cf2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 09:43:16 +0000 Subject: [PATCH 2/7] Add entities display & UX spec; cross-link from KB spec and README --- README.md | 1 + docs/entities-display.md | 127 ++++++++++++++++++++++++++++++++ docs/entities-knowledge-base.md | 2 + 3 files changed, 130 insertions(+) create mode 100644 docs/entities-display.md diff --git a/README.md b/README.md index 233a42b..e80a99a 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ This repo contains: - [How verification works](docs/how-verify-works.md) — normalization pipeline and comparison logic - [Verification report](docs/verification-report.md) — error categories found and lessons learned - [Entities knowledge base](docs/entities-knowledge-base.md) — design spec for an accumulating store of people, places, and plants in the Mishnah (v0, foundational) + - [Entities display & UX](docs/entities-display.md) — how entities are rendered: the in-text color overlay and the derived entity/map/gallery views (v0, foundational) - **Scripts** — full pipeline for downloading, formatting, verifying, fixing, and merging: - `scripts/download.py` — fetch raw JSON from Sefaria API - `scripts/format.py` — format JSON using Ollama, Anthropic API, or Claude Code diff --git a/docs/entities-display.md b/docs/entities-display.md new file mode 100644 index 0000000..25c1ed6 --- /dev/null +++ b/docs/entities-display.md @@ -0,0 +1,127 @@ +# Mishnah Entities — Display & UX Spec + +**Status:** v0 (foundational). This spec defines how the people, places, and plants in the [entities knowledge base](entities-knowledge-base.md) are *displayed* to a reader — the rendered views and the in-text overlay. It is deliberately separate from the KB spec: the knowledge base is the source of truth and knows nothing about presentation; everything here is a **derived view** on top of it. + +It builds on two existing repo conventions: the masechet HTML structure and deep-link anchors in [`docs/html-format.md`](html-format.md), and the word-normalization pipeline in [`docs/how-verify-works.md`](how-verify-works.md), which phase 3 reuses. + +----- + +## 1. Principle: the canonical text is never touched + +The committed `masechot/*.html` files are the canonical reading artifacts. **The display layer never edits them.** All entity presentation is produced into *separate, derived, regenerable* artifacts (gitignored, like `knowledge.db`, `sefaria/`, and `output/`). Re-running the build reproduces them exactly; deleting them loses nothing. + +This keeps the house style (`docs/editorial-style.md`) and word-fidelity guarantee (`verify.py`) completely insulated from anything we do for entities. + +----- + +## 2. Two visual states + +The enriched reading view has two states: + +1. **Default — invisible.** It reads identically to the canonical masechet. No color, no underlines, no icons in the running text. A reader who never touches the controls sees the pure text. +2. **Overlay on — quiet color.** Toggling brings up a color overlay on entity mentions: + + | Type | Color | Icon | + | --- | --- | --- | + | Plants | green | 🌿 | + | Places | red | 📍 | + | People | blue | 👤 | + + Each colored mention is a link to that entity's page (§5). Toggles are **per-type and independent** — a reader studying *Kilayim* can light up only plants; one reading *Avot* can light up only people. + +The icons are used in the toggle controls, the legend, and on entity pages. The *inline* overlay is color-only by default (see the accessibility note in §3). + +----- + +## 3. Color & accessibility + +The palette is green / red / blue as above. **Known issue, recorded for follow-up:** red (places) and green (plants) collide for red-green color vision deficiency (~8% of men), and the inline overlay is the one surface that carries color *without* an accompanying icon. + +Mitigations, ranked, to decide before any public release (not a blocker for dev — dev uses the palette as described): + +1. **Distinct underline styles per type** (e.g. plants dotted, places dashed, people solid) layered under the color, so the cue survives without color. +2. **A small leading icon** on inline mentions — reuses 🌿/📍/👤, but adds visual weight to the running text. +3. **A colorblind-safe palette** swap (e.g. blue / orange / teal). + +Everywhere *outside* the running text (toggles, legend, entity pages), the per-type icon already provides a non-color channel, so those surfaces are fine as-is. + +----- + +## 4. Build pipeline — three phases, each a view on the one below + +``` +data/**.yaml ──kb-build──▶ knowledge.db (1) source of truth → SQLite index +knowledge.db ──kb-render─▶ entity pages (2) one HTML page per entity + + map / flora-gallery / index (pure DB views; no mishna text) +masechot/*.html + DB ──kb-enrich─▶ enriched masechot (3) mark mishna mentions in the text, + link each to its (2) page +``` + +The order matters and is intentional: + +- **Phase 1 — `scripts/kb-build.py`** compiles the YAML into `knowledge.db` (per the KB spec §11). Nothing here is presentational. +- **Phase 2 — `scripts/kb-render.py`** generates the standalone, DB-only views: one **entity page** per person/place/plant (§5), plus aggregate views — a **places map**, a **flora gallery**, and a **who's-who / index**. These never contain mishna text; they are pure projections of the KB. +- **Phase 3 — `scripts/kb-enrich.py`** is the *join*, and therefore last: it takes a canonical masechet plus the DB and produces an **enriched copy** in which entity mentions in the running text are wrapped, colored (under the overlay), and linked to the phase-2 entity pages. It depends on phase 2 (the link targets must exist) and on the appearance linkage (to know which mishnayot mention what). + +All phase-2 and phase-3 outputs are derived artifacts: gitignored, regenerable, output to a build/publish directory rather than committed over `masechot/`. + +----- + +## 5. Entity pages — the canonical info surface + +Clicking an entity opens its **own page** (a real page in a new tab — the "popup"), not just a tooltip. This is deliberate: the same page is what the map, gallery, and index link to, so each entity has **one canonical surface**, and that surface has room a tooltip never would. + +Each entity page carries, by type: + +- **People** — names (he/en/variants), generation & floruit, relationships (teachers/students, as links to their pages), bio claims. +- **Places** — a map at the displayed coordinate, photo, region; classical/modern names. +- **Plants** — image/illustration, display taxon, botanical/common names. + +And for **all** types, the big payoff a tooltip can't give: **every appearance across all 63 masechot**, as deep links into the enriched text via the existing anchor scheme (`masechot/{slug}.html#mishna-N-M`). + +### Contested facts must stay visible + +The KB keeps dissent on purpose (multiple plant candidates, disputed coordinates); the entity page must not flatten it: + +- **Plants** — show the display taxon prominently, then list the other candidates with their source and confidence (e.g. *identified as Origanum syriacum (Feliks); also proposed: Satureja thymbra (minority)*). +- **Places** — when `consensus` is `disputed`, show it as disputed and surface the competing coordinates/sources rather than silently picking one pin. + +----- + +## 6. Phase 3 — anchoring entity words (the keystone, and the finicky part) + +Appearances in the KB are **mishna-granular** (`shekalim 4:2`); the overlay needs the **exact word(s)** to color and link. Resolution (chosen approach, revisable): + +- `kb-enrich` **matches surface forms** drawn from the entity's `names` / `variants` against the mishna's running text. +- It does so through **`verify.py`'s normalization pipeline** — nikkud stripping, final-letter folding, double-vav normalization (`docs/how-verify-works.md`) — so a match survives vowelization and spelling variance. Reusing that pipeline means the matcher is built on logic the project already trusts. + +This keeps span-finding out of the detector (which stays mishna-granular). It is expected to be the **finicky** part — inflection, construct forms, and multi-word terms will need iteration — and is the natural first place to invest once the views exist. If surface matching proves too lossy, the fallback is to have the detector emit character offsets; we deliberately defer that cost until proven necessary. + +----- + +## 7. Ambiguous matches (open design, for phase 3) + +A surface form can legitimately resolve to **more than one** entity (two places sharing a name; a word that is both a plant term and a name). Both candidate entities already exist as phase-2 pages, so a marked mention can become a **choice between them** rather than a single link. Direction, to be finalized in phase 3: + +- **Mark only when resolvable.** If the appearance for that mishna disambiguates the mention, link straight to the right page. **Never guess** — an unmarked word is better than a wrong link. +- **When genuinely ambiguous, present the choice rather than suppressing it.** Options to weigh: a small disambiguation chooser (a popover listing the candidate pages), or a distinct "ambiguous" styling that links to a tiny chooser surface. Since both targets exist in the rendered set, the link is naturally "this or that." + +This is recorded as an open item; the matcher in §6 and this section will be designed together. + +----- + +## 8. Implementation notes + +- **JS-free reading view.** The overlay toggle is **CSS-only** (a hidden checkbox per type + `:checked ~` rules flipping the span colors), so the enriched masechot stay static, offline, and printable. The only interactive surfaces are the entity pages themselves (and even those can start as plain pages). +- **No markup in the canonical files.** Wrapping happens only in the enriched copies. (For the record, this is safe regardless: `verify.py` strips all tags before word comparison, so entity spans never affect fidelity — but we keep the canonical files clean as a matter of principle.) +- **Scripts** live in the shared `scripts/` directory alongside the rest of the pipeline: `kb-render.py` (phase 2), `kb-enrich.py` (phase 3). +- **Artifacts** (entity pages, map, gallery, index, enriched masechot) are derived and gitignored. + +----- + +## 9. Open items + +1. **Colorblind-safe treatment of the inline overlay** (§3) — pick a mitigation before public release. +2. **Word anchoring** (§6) — surface-form matching is expected to need iteration; detector offsets remain a fallback. +3. **Ambiguous matches** (§7) — finalize the chooser vs. distinct-styling approach. +4. **Map / gallery / index detail** — layout of the aggregate phase-2 views is not yet specified. diff --git a/docs/entities-knowledge-base.md b/docs/entities-knowledge-base.md index 88e28ab..84dab13 100644 --- a/docs/entities-knowledge-base.md +++ b/docs/entities-knowledge-base.md @@ -4,6 +4,8 @@ This is a companion subsystem to the text-formatting pipeline that produces `masechot/`. It is adapted to this repo's existing conventions: the slug rules already used for masechet filenames, the deep-link anchor scheme in [`docs/html-format.md`](html-format.md), the practice of gitignoring derived artifacts, and a flat `scripts/` directory. Where this spec departs from a generic "fresh repo" layout, it is to match those conventions; the departures are called out inline. +How these entities are *displayed* to a reader — the in-text overlay and the rendered entity/map/gallery views — is deliberately **not** defined here; it is a derived view, specified separately in [`docs/entities-display.md`](entities-display.md). + ----- ## 1. What this is, and what it is not From 0e15cbaa60f65096b04bddebce2ceb4d7062ccef Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 10:01:03 +0000 Subject: [PATCH 3/7] Phase 1: entities KB schemas, registries, exemplars, and validator - entities/schema: claim, source, person, place, plant JSON Schemas - entities/data: sources.yaml, vocab (sedarim/generations/halachic/regions), and three validating exemplars (akiva, tzippori, chitah) - scripts/kb-validate.py: JSON Schema + semantic cross-checks (source existence, canonical appearance slugs, filename<->slug) - pre-commit hook + GitHub Actions CI; gitignore derived knowledge.db/conflicts.log --- .github/workflows/kb-validate.yml | 24 +++ .gitignore | 4 + .pre-commit-config.yaml | 12 ++ README.md | 1 + entities/data/people/akiva.yaml | 30 +++ entities/data/places/tzippori.yaml | 31 ++++ entities/data/plants/chitah.yaml | 33 ++++ entities/data/sources.yaml | 45 +++++ entities/data/vocab/generations.yaml | 9 + entities/data/vocab/halachic-categories.yaml | 21 +++ entities/data/vocab/regions.yaml | 14 ++ entities/data/vocab/sedarim.yaml | 81 ++++++++ entities/requirements.txt | 3 + entities/schema/claim.schema.json | 39 ++++ entities/schema/person.schema.json | 88 +++++++++ entities/schema/place.schema.json | 105 +++++++++++ entities/schema/plant.schema.json | 115 ++++++++++++ entities/schema/source.schema.json | 40 ++++ scripts/kb-validate.py | 183 +++++++++++++++++++ 19 files changed, 878 insertions(+) create mode 100644 .github/workflows/kb-validate.yml create mode 100644 .pre-commit-config.yaml create mode 100644 entities/data/people/akiva.yaml create mode 100644 entities/data/places/tzippori.yaml create mode 100644 entities/data/plants/chitah.yaml create mode 100644 entities/data/sources.yaml create mode 100644 entities/data/vocab/generations.yaml create mode 100644 entities/data/vocab/halachic-categories.yaml create mode 100644 entities/data/vocab/regions.yaml create mode 100644 entities/data/vocab/sedarim.yaml create mode 100644 entities/requirements.txt create mode 100644 entities/schema/claim.schema.json create mode 100644 entities/schema/person.schema.json create mode 100644 entities/schema/place.schema.json create mode 100644 entities/schema/plant.schema.json create mode 100644 entities/schema/source.schema.json create mode 100644 scripts/kb-validate.py diff --git a/.github/workflows/kb-validate.yml b/.github/workflows/kb-validate.yml new file mode 100644 index 0000000..2eeb872 --- /dev/null +++ b/.github/workflows/kb-validate.yml @@ -0,0 +1,24 @@ +name: kb-validate + +on: + push: + paths: + - "entities/**" + - "scripts/kb-validate.py" + - ".github/workflows/kb-validate.yml" + pull_request: + paths: + - "entities/**" + - "scripts/kb-validate.py" + - ".github/workflows/kb-validate.yml" + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - run: pip install -r entities/requirements.txt + - run: python3 scripts/kb-validate.py diff --git a/.gitignore b/.gitignore index 096f279..7bb0a41 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,7 @@ __pycache__/ sefaria/ output/ + +# Entities KB derived artifacts (rebuilt from entities/data/ — never committed) +entities/knowledge.db +entities/conflicts.log diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..291254a --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,12 @@ +# Validation is the contract: a file that does not validate does not land. +# Install once with `pip install pre-commit && pre-commit install`. +repos: + - repo: local + hooks: + - id: kb-validate + name: validate entities knowledge base + entry: python scripts/kb-validate.py + language: python + additional_dependencies: ["PyYAML>=6.0", "jsonschema>=4.18"] + files: ^entities/(data|schema)/.*\.(ya?ml|json)$ + pass_filenames: false diff --git a/README.md b/README.md index e80a99a..deb108d 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ This repo contains: - `scripts/fix.py` — programmatic fixes + LLM regen for errors - `scripts/merge.py` — apply JSON corrections into HTML files - `scripts/update-readme.py` — regenerate masechot table from meta tags +- **Entities knowledge base** (`entities/`) — an accumulating store of people, places, and plants in the Mishnah ([KB spec](docs/entities-knowledge-base.md), [display spec](docs/entities-display.md)). `scripts/kb-validate.py` validates `entities/data/` against the JSON schemas (run `pip install -r entities/requirements.txt` first). - **A Claude skill** (`.claude/skills/mishnah/`) for interactive formatting in Claude Code ## The Style diff --git a/entities/data/people/akiva.yaml b/entities/data/people/akiva.yaml new file mode 100644 index 0000000..ccb4ed5 --- /dev/null +++ b/entities/data/people/akiva.yaml @@ -0,0 +1,30 @@ +slug: akiva +status: enriched +type: tanna +aliases: [] +names: + he: "רבי עקיבא" + en: "Rabbi Akiva" + variants: ["רבי עקיבא בן יוסף", "ר' עקיבא"] +era: + generation: 3 # tannaitic generation — see vocab/generations.yaml + floruit: "early 2nd c. CE" + locale: ["bnei-brak"] # slug into places/ (need not exist yet) +relationships: # borrowed; disputants/colleagues are DERIVED at build time + teachers: ["eliezer-b-hyrcanus", "yehoshua-b-hananiah"] + students: ["meir", "yehuda-b-ilai", "shimon-b-yochai"] + family: [] +bio: + - value: "Leading tanna of the 3rd generation; foundational to the structure of the Mishnah." + source: wikidata + confidence: accepted + date: "2026-06-14" + confirmed: false +appearances: # detector-populated later; a few real ones shown as fixtures + mishnah: ["brachot 9:5", "makkot 1:10", "avot 3:13"] + other: ["Tosefta Berakhot 4:16"] +ids: + wikidata_qid: Q310357 + sefaria_slug: rabbi-akiva + hyman_ref: null + bonayich_id: null diff --git a/entities/data/places/tzippori.yaml b/entities/data/places/tzippori.yaml new file mode 100644 index 0000000..3da5f16 --- /dev/null +++ b/entities/data/places/tzippori.yaml @@ -0,0 +1,31 @@ +slug: tzippori +status: enriched +type: settlement +aliases: [] +names: + he: "ציפורי" + en: "Tzippori" + variants: ["צפורי"] + classical: ["Sepphoris", "Diocaesarea"] + modern: { he: "ציפורי", ar: "صفورية", en: "Tzippori" } +geo: + coordinates: # display value plus retained dissent (one candidate here) + - value: { lat: 32.7522, lon: 35.2797 } + source: pleiades + confidence: accepted + confirmed: false + region: galilee # see vocab/regions.yaml + modern_admin: "Israel, Northern District" + consensus: identified +media: + photo: "https://commons.wikimedia.org/wiki/Category:Sepphoris" + map_link: "https://www.openstreetmap.org/?mlat=32.7522&mlon=35.2797" +plan: null # settlements have no temple plan +appearances: # detector-populated later + mishnah: [] + other: [] +ids: + wikidata_qid: Q745966 + pleiades_id: "678378" + geonames_id: null + sefaria_slug: null diff --git a/entities/data/plants/chitah.yaml b/entities/data/plants/chitah.yaml new file mode 100644 index 0000000..692b566 --- /dev/null +++ b/entities/data/plants/chitah.yaml @@ -0,0 +1,33 @@ +slug: chitah +status: enriched +term: + he: "חיטה" + variants: ["חטה", "חטים", "חיטים"] # spelling + plural forms + en_common: "wheat" +term_type: species +identification: + consensus: identified # an uncontested term carries a single candidate; + display_taxon: "Triticum aestivum" # a contested term (e.g. ezov) would carry >1 + candidates: + - value: { taxon: "Triticum aestivum", rank: species } + ids: { wikidata_qid: Q12100, gbif_id: null, powo_id: null } + source: feliks + asserted_by: Feliks + confidence: accepted + confirmed: false +names: + botanical: "Triticum aestivum" + family: "Poaceae" + common_en: ["bread wheat", "wheat"] + modern_he: ["חיטה"] + arabic: ["قمح"] +usage: + ethnobotanical: ["food"] + halachic: ["kilayim", "maaser-dagan", "challah", "chadash"] # DERIVED; see vocab/halachic-categories.yaml +media: + photo: "https://commons.wikimedia.org/wiki/Category:Triticum_aestivum" + illustration: "https://commons.wikimedia.org/wiki/Category:Köhler%27s_Medizinal-Pflanzen" +appearances: # detector-populated later; a few real ones shown as fixtures + mishnah: ["kilayim 1:1", "challah 1:1", "pesachim 2:5"] + other: [] +ids: {} # term-level external ids rare; real ids live on candidates diff --git a/entities/data/sources.yaml b/entities/data/sources.yaml new file mode 100644 index 0000000..c5e991b --- /dev/null +++ b/entities/data/sources.yaml @@ -0,0 +1,45 @@ +# Source registry. Every claim's `source:` references a key here. +# trust_tier: lower number == higher trust (0 == manual curation). +# See docs/entities-knowledge-base.md §6. + +sefaria: + citation: "Sefaria, sefaria.org" + type: structured + license: public-domain # the Mishnah text itself is PD; same source the text pipeline uses + trust_tier: 1 # canonical reference/appearance source for this project + +wikidata: + citation: "Wikidata, wikidata.org" + type: structured + license: CC0 + trust_tier: 2 + +pleiades: + citation: "Pleiades: A Gazetteer of Past Places, pleiades.stoa.org" + type: structured + license: CC-BY + trust_tier: 1 + +low: + citation: "Immanuel Löw, Die Flora der Juden, 4 vols., Vienna 1924–1934" + type: scholarly-prose + license: public-domain # author d. 1944; pre-1929 vols PD, life+70 elapsed + trust_tier: 1 + +feliks: + citation: "Yehuda Feliks, works on flora of rabbinic literature" + type: scholarly-prose + license: in-copyright # reference/adjudicate only — do NOT bulk-redistribute + trust_tier: 1 + +detector: + citation: "Mishnah entity detector (this project)" + type: derived + license: project + trust_tier: 3 + +manual: + citation: "Manual curation" + type: human + license: project + trust_tier: 0 diff --git a/entities/data/vocab/generations.yaml b/entities/data/vocab/generations.yaml new file mode 100644 index 0000000..19885dc --- /dev/null +++ b/entities/data/vocab/generations.yaml @@ -0,0 +1,9 @@ +# Tannaitic generations — the reliable time unit for people (person.era.generation). +# Approximate floruit windows; conventional scholarly numbering 1–6. + +1: { floruit: "c. 10–80 CE", note: "Destruction generation; Yavne founders (Yochanan b. Zakkai, Gamliel the Elder)." } +2: { floruit: "c. 80–110 CE", note: "Yavne (Gamliel II, Eliezer b. Hyrcanus, Yehoshua b. Hananiah)." } +3: { floruit: "c. 110–135 CE", note: "Pre-/Bar-Kokhba (Akiva, Tarfon, Yishmael)." } +4: { floruit: "c. 135–170 CE", note: "Usha (Meir, Yehuda b. Ilai, Shimon b. Yochai, Yose)." } +5: { floruit: "c. 170–200 CE", note: "Redaction generation (Yehuda ha-Nasi and colleagues)." } +6: { floruit: "c. 200–220 CE", note: "Transitional tannaim, contemporaries of the early amoraim." } diff --git a/entities/data/vocab/halachic-categories.yaml b/entities/data/vocab/halachic-categories.yaml new file mode 100644 index 0000000..5513845 --- /dev/null +++ b/entities/data/vocab/halachic-categories.yaml @@ -0,0 +1,21 @@ +# Controlled vocabulary of agricultural/halachic categories used on plant terms +# (plant.usage.halachic). Derived from the Mishnah. Slugs; some coincide with +# masechet slugs (kilayim, challah, orlah, bikkurim) but here denote the category. + +kilayim: "Forbidden mixtures (diverse seeds/grafts)." +terumah: "Priestly heave-offering." +maaser: "Tithe (general)." +maaser-dagan: "Tithe of grain (dagan)." +maaser-rishon: "First tithe (to the Levite)." +maaser-sheni: "Second tithe (eaten in Jerusalem)." +maaser-ani: "Poor man's tithe." +challah: "Dough-offering separated from bread." +orlah: "Fruit of a tree's first three years, forbidden." +chadash: "New grain, forbidden before the Omer." +sheviit: "Produce of the sabbatical year." +peah: "Corner of the field left for the poor." +leket: "Gleanings left for the poor." +shichechah: "Forgotten sheaf left for the poor." +bikkurim: "First fruits brought to the Temple." +demai: "Produce of uncertain tithing status." +neta-revai: "Fourth-year fruit, eaten in Jerusalem." diff --git a/entities/data/vocab/regions.yaml b/entities/data/vocab/regions.yaml new file mode 100644 index 0000000..15ca7a0 --- /dev/null +++ b/entities/data/vocab/regions.yaml @@ -0,0 +1,14 @@ +# Controlled vocabulary of regions used on places (place.geo.region). Slugs. + +galilee: "The Galilee (Lower and Upper)." +golan: "The Golan / Gaulanitis." +jezreel-valley: "The Jezreel Valley." +coastal-plain: "The Mediterranean coastal plain (incl. Sharon)." +shephelah: "The Shephelah (Judean lowlands)." +judea: "Judea / the Judean hills." +jerusalem: "Jerusalem and its environs." +samaria: "Samaria / the central highlands." +transjordan: "Transjordan (Peraea, the eastern bank)." +negev: "The Negev / southern arid zone." +dead-sea: "The Dead Sea basin." +diaspora: "Outside the Land of Israel (Bavel, Egypt, Rome, etc.)." diff --git a/entities/data/vocab/sedarim.yaml b/entities/data/vocab/sedarim.yaml new file mode 100644 index 0000000..719f81b --- /dev/null +++ b/entities/data/vocab/sedarim.yaml @@ -0,0 +1,81 @@ +# Canonical masechet vocabulary, grouped by seder. +# `slug` matches the masechot/*.html filenames exactly — appearance refs +# (`{slug} {chapter}:{mishna}`) are validated against this list, and resolve +# to masechot/{slug}.html#mishna-N-M via the anchor scheme in docs/html-format.md. +# `sefaria` is the Sefaria API name; `chapters` is the perek count. +# Mirrors scripts/download.py SEDARIM + scripts/verify.py MASECHET_FILENAMES. + +Zeraim: + - { slug: brachot, sefaria: Berakhot, chapters: 9 } + - { slug: peah, sefaria: Peah, chapters: 8 } + - { slug: demai, sefaria: Demai, chapters: 7 } + - { slug: kilayim, sefaria: Kilayim, chapters: 9 } + - { slug: sheviit, sefaria: Sheviit, chapters: 10 } + - { slug: terumot, sefaria: Terumot, chapters: 11 } + - { slug: maaserot, sefaria: Maasrot, chapters: 5 } + - { slug: maaser-sheni, sefaria: Maaser_Sheni, chapters: 5 } + - { slug: challah, sefaria: Challah, chapters: 4 } + - { slug: orlah, sefaria: Orlah, chapters: 3 } + - { slug: bikkurim, sefaria: Bikkurim, chapters: 4 } + +Moed: + - { slug: shabbat, sefaria: Shabbat, chapters: 24 } + - { slug: eruvin, sefaria: Eruvin, chapters: 10 } + - { slug: pesachim, sefaria: Pesachim, chapters: 10 } + - { slug: shekalim, sefaria: Shekalim, chapters: 8 } + - { slug: yoma, sefaria: Yoma, chapters: 8 } + - { slug: sukkah, sefaria: Sukkah, chapters: 5 } + - { slug: beitzah, sefaria: Beitzah, chapters: 5 } + - { slug: rosh-hashanah, sefaria: Rosh_Hashanah, chapters: 4 } + - { slug: taanit, sefaria: Taanit, chapters: 4 } + - { slug: megillah, sefaria: Megillah, chapters: 4 } + - { slug: moed-katan, sefaria: Moed_Katan, chapters: 3 } + - { slug: chagigah, sefaria: Chagigah, chapters: 3 } + +Nashim: + - { slug: yevamot, sefaria: Yevamot, chapters: 16 } + - { slug: ketubot, sefaria: Ketubot, chapters: 13 } + - { slug: nedarim, sefaria: Nedarim, chapters: 11 } + - { slug: nazir, sefaria: Nazir, chapters: 9 } + - { slug: sotah, sefaria: Sotah, chapters: 9 } + - { slug: gittin, sefaria: Gittin, chapters: 9 } + - { slug: kiddushin, sefaria: Kiddushin, chapters: 4 } + +Nezikin: + - { slug: bava-kamma, sefaria: Bava_Kamma, chapters: 10 } + - { slug: bava-metzia, sefaria: Bava_Metzia, chapters: 10 } + - { slug: bava-batra, sefaria: Bava_Batra, chapters: 10 } + - { slug: sanhedrin, sefaria: Sanhedrin, chapters: 11 } + - { slug: makkot, sefaria: Makkot, chapters: 3 } + - { slug: shevuot, sefaria: Shevuot, chapters: 8 } + - { slug: eduyot, sefaria: Eduyot, chapters: 8 } + - { slug: avodah-zarah, sefaria: Avodah_Zarah, chapters: 5 } + - { slug: avot, sefaria: Avot, chapters: 6 } + - { slug: horayot, sefaria: Horayot, chapters: 3 } + +Kodashim: + - { slug: zevachim, sefaria: Zevachim, chapters: 14 } + - { slug: menachot, sefaria: Menachot, chapters: 13 } + - { slug: chullin, sefaria: Chullin, chapters: 12 } + - { slug: bekhorot, sefaria: Bekhorot, chapters: 9 } + - { slug: arakhin, sefaria: Arakhin, chapters: 9 } + - { slug: temurah, sefaria: Temurah, chapters: 7 } + - { slug: keritot, sefaria: Keritot, chapters: 6 } + - { slug: meilah, sefaria: Meilah, chapters: 6 } + - { slug: tamid, sefaria: Tamid, chapters: 7 } + - { slug: middot, sefaria: Middot, chapters: 5 } + - { slug: kinnim, sefaria: Kinnim, chapters: 3 } + +Taharot: + - { slug: keilim, sefaria: Kelim, chapters: 30 } + - { slug: ohalot, sefaria: Ohalot, chapters: 18 } + - { slug: negaim, sefaria: Negaim, chapters: 14 } + - { slug: parah, sefaria: Parah, chapters: 12 } + - { slug: taharot, sefaria: Taharot, chapters: 10 } + - { slug: mikvaot, sefaria: Mikvaot, chapters: 10 } + - { slug: niddah, sefaria: Niddah, chapters: 10 } + - { slug: makhshirin, sefaria: Makhshirin, chapters: 6 } + - { slug: zavim, sefaria: Zavim, chapters: 5 } + - { slug: tevul-yom, sefaria: Tevul_Yom, chapters: 4 } + - { slug: yadayim, sefaria: Yadayim, chapters: 4 } + - { slug: uktzin, sefaria: Oktzin, chapters: 3 } diff --git a/entities/requirements.txt b/entities/requirements.txt new file mode 100644 index 0000000..977c4e2 --- /dev/null +++ b/entities/requirements.txt @@ -0,0 +1,3 @@ +# Dependencies for the entities knowledge base tooling (scripts/kb-*.py). +PyYAML>=6.0 +jsonschema>=4.18 diff --git a/entities/schema/claim.schema.json b/entities/schema/claim.schema.json new file mode 100644 index 0000000..8ee7231 --- /dev/null +++ b/entities/schema/claim.schema.json @@ -0,0 +1,39 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "claim.schema.json", + "title": "Claim", + "description": "A single sourced, contestable assertion. The shared spine of the KB: plant identification candidates, place coordinates, biographical facts, etc. are all claims. See docs/entities-knowledge-base.md §5. Extra keys are permitted so specialized claims may carry more (e.g. `ids` on a plant candidate, `provenance` on a medicinal use).", + "type": "object", + "required": ["value", "source"], + "properties": { + "value": { + "description": "The asserted value — scalar or object. Intentionally unconstrained." + }, + "source": { + "type": "string", + "minLength": 1, + "description": "Key into data/sources.yaml. Required — no source, no claim." + }, + "confidence": { + "enum": ["accepted", "probable", "minority", "disputed", "unknown"] + }, + "asserted_by": { + "type": "string", + "description": "Scholar or authority behind the claim, e.g. Feliks, Löw." + }, + "date": { + "type": "string", + "pattern": "^\\d{4}-\\d{2}-\\d{2}$", + "description": "ISO date (YYYY-MM-DD) the claim was recorded." + }, + "confirmed": { + "type": "boolean", + "default": false, + "description": "true == human-ratified; never auto-overwritten (KB spec §8)." + }, + "note": { + "type": "string", + "description": "Free-text adjudication reasoning." + } + } +} diff --git a/entities/schema/person.schema.json b/entities/schema/person.schema.json new file mode 100644 index 0000000..831495d --- /dev/null +++ b/entities/schema/person.schema.json @@ -0,0 +1,88 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "person.schema.json", + "title": "Person", + "description": "A person in the Mishnah. See docs/entities-knowledge-base.md §9.1.", + "type": "object", + "required": ["slug", "status", "type", "names"], + "additionalProperties": false, + "properties": { + "slug": { "$ref": "#/$defs/slug" }, + "status": { "$ref": "#/$defs/status" }, + "type": { "enum": ["tanna", "amora", "biblical", "named-layperson", "collective"] }, + "aliases": { + "type": "array", + "items": { "$ref": "#/$defs/slug" }, + "default": [] + }, + "names": { + "type": "object", + "required": ["he"], + "additionalProperties": false, + "properties": { + "he": { "type": "string", "minLength": 1 }, + "en": { "type": "string" }, + "variants": { "type": "array", "items": { "type": "string" } } + } + }, + "era": { + "type": "object", + "additionalProperties": false, + "properties": { + "generation": { + "type": "integer", + "minimum": 1, + "maximum": 6, + "description": "Tannaitic generation 1–6 — the reliable time unit." + }, + "floruit": { "type": "string" }, + "locale": { + "type": "array", + "items": { "$ref": "#/$defs/slug" }, + "description": "Slugs into places/." + } + } + }, + "relationships": { + "type": "object", + "additionalProperties": false, + "description": "Borrowed (Wikidata/biographical) typed edges. Disputants/colleagues are DERIVED at build time, not stored.", + "properties": { + "teachers": { "type": "array", "items": { "$ref": "#/$defs/slug" } }, + "students": { "type": "array", "items": { "$ref": "#/$defs/slug" } }, + "family": { "type": "array", "items": { "$ref": "#/$defs/slug" } } + } + }, + "bio": { "type": "array", "items": { "$ref": "claim.schema.json" } }, + "appearances": { "$ref": "#/$defs/appearances" }, + "ids": { + "type": "object", + "additionalProperties": false, + "properties": { + "wikidata_qid": { "type": ["string", "null"] }, + "sefaria_slug": { "type": ["string", "null"] }, + "hyman_ref": { "type": ["string", "null"] }, + "bonayich_id": { "type": ["string", "null"] } + } + } + }, + "$defs": { + "slug": { "type": "string", "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$" }, + "status": { "enum": ["stub", "enriched", "reviewed", "confirmed"] }, + "appearances": { + "type": "object", + "additionalProperties": false, + "properties": { + "mishnah": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)* \\d+:\\d+$", + "description": "Canonical form `{masechet-slug} {chapter}:{mishna}`, e.g. `shekalim 4:2`." + } + }, + "other": { "type": "array", "items": { "type": "string" } } + } + } + } +} diff --git a/entities/schema/place.schema.json b/entities/schema/place.schema.json new file mode 100644 index 0000000..259e24b --- /dev/null +++ b/entities/schema/place.schema.json @@ -0,0 +1,105 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "place.schema.json", + "title": "Place", + "description": "A place in the Mishnah. See docs/entities-knowledge-base.md §9.2.", + "type": "object", + "required": ["slug", "status", "type", "names"], + "additionalProperties": false, + "properties": { + "slug": { "$ref": "#/$defs/slug" }, + "status": { "$ref": "#/$defs/status" }, + "type": { "enum": ["settlement", "region", "temple_structure", "water_feature", "legal_domain"] }, + "aliases": { + "type": "array", + "items": { "$ref": "#/$defs/slug" }, + "default": [] + }, + "names": { + "type": "object", + "required": ["he"], + "additionalProperties": false, + "properties": { + "he": { "type": "string", "minLength": 1 }, + "en": { "type": "string" }, + "variants": { "type": "array", "items": { "type": "string" } }, + "classical": { + "type": "array", + "items": { "type": "string" }, + "description": "Greco-Roman toponym; settlements only, often empty." + }, + "modern": { + "type": "object", + "additionalProperties": false, + "properties": { + "he": { "type": "string" }, + "ar": { "type": "string" }, + "en": { "type": "string" } + } + } + } + }, + "geo": { + "type": ["object", "null"], + "additionalProperties": false, + "description": "null for temple_structure & legal_domain.", + "properties": { + "coordinates": { + "type": "array", + "items": { "$ref": "claim.schema.json" }, + "description": "Claims whose value is { lat, lon }. The displayed value plus retained dissent." + }, + "region": { "type": "string" }, + "modern_admin": { "type": "string" }, + "consensus": { "enum": ["identified", "probable", "disputed", "unknown"] } + } + }, + "media": { + "type": "object", + "additionalProperties": false, + "properties": { + "photo": { "type": ["string", "null"] }, + "map_link": { "type": ["string", "null"] } + } + }, + "plan": { + "type": ["object", "null"], + "additionalProperties": false, + "description": "temple_structure only.", + "properties": { + "position": { "type": "string" }, + "source_ref": { "type": "string" }, + "diagram_ref": { "type": "string" } + } + }, + "appearances": { "$ref": "#/$defs/appearances" }, + "ids": { + "type": "object", + "additionalProperties": false, + "properties": { + "wikidata_qid": { "type": ["string", "null"] }, + "pleiades_id": { "type": ["string", "null"] }, + "geonames_id": { "type": ["string", "null"] }, + "sefaria_slug": { "type": ["string", "null"] } + } + } + }, + "$defs": { + "slug": { "type": "string", "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$" }, + "status": { "enum": ["stub", "enriched", "reviewed", "confirmed"] }, + "appearances": { + "type": "object", + "additionalProperties": false, + "properties": { + "mishnah": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)* \\d+:\\d+$" + } + }, + "other": { "type": "array", "items": { "type": "string" } } + } + } + } +} diff --git a/entities/schema/plant.schema.json b/entities/schema/plant.schema.json new file mode 100644 index 0000000..43ce889 --- /dev/null +++ b/entities/schema/plant.schema.json @@ -0,0 +1,115 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "plant.schema.json", + "title": "Plant", + "description": "A plant term in the Mishnah. The primary key is the Hebrew term, not a taxon — identifications are contested claims. See docs/entities-knowledge-base.md §9.3.", + "type": "object", + "required": ["slug", "status", "term", "term_type"], + "additionalProperties": false, + "properties": { + "slug": { "$ref": "#/$defs/slug" }, + "status": { "$ref": "#/$defs/status" }, + "term": { + "type": "object", + "required": ["he"], + "additionalProperties": false, + "properties": { + "he": { "type": "string", "minLength": 1 }, + "variants": { + "type": "array", + "items": { "type": "string" }, + "description": "Spelling + plural + inflected forms." + }, + "en_common": { "type": "string" } + } + }, + "term_type": { "enum": ["species", "genus_or_group", "folk_category", "product", "plant_part"] }, + "identification": { + "type": "object", + "additionalProperties": false, + "properties": { + "consensus": { "enum": ["identified", "contested", "unknown"] }, + "display_taxon": { + "type": "string", + "description": "The chosen taxon for rendering. Absent for folk_category terms (use `members`)." + }, + "members": { + "type": "array", + "items": { "type": "string" }, + "description": "folk_category only: the set of member taxa, in place of a single display_taxon." + }, + "candidates": { + "type": "array", + "items": { "$ref": "claim.schema.json" }, + "description": "Identification claims. A contested term carries >1. Each value is { taxon, rank }; external ids live on the candidate, not the term." + } + } + }, + "names": { + "type": "object", + "additionalProperties": false, + "description": "Keyed to display_taxon; from botanical DBs.", + "properties": { + "botanical": { "type": "string" }, + "family": { "type": "string" }, + "common_en": { "type": "array", "items": { "type": "string" } }, + "modern_he": { "type": "array", "items": { "type": "string" } }, + "arabic": { "type": "array", "items": { "type": "string" } } + } + }, + "usage": { + "type": "object", + "additionalProperties": false, + "properties": { + "ethnobotanical": { "type": "array", "items": { "type": "string" } }, + "medicinal": { + "type": "array", + "items": { "$ref": "claim.schema.json" }, + "description": "Historical/ethnobotanical, NOT medical guidance; provenance-tagged claims." + }, + "halachic": { + "type": "array", + "items": { "type": "string" }, + "description": "DERIVED from the Mishnah. Keys into data/vocab/halachic-categories.yaml." + } + } + }, + "media": { + "type": "object", + "additionalProperties": false, + "properties": { + "photo": { "type": ["string", "null"] }, + "illustration": { "type": ["string", "null"] } + } + }, + "appearances": { "$ref": "#/$defs/appearances" }, + "ids": { + "type": "object", + "description": "Term-level external ids are rare; real ids live on identification candidates.", + "additionalProperties": false, + "properties": { + "wikidata_qid": { "type": ["string", "null"] }, + "gbif_id": { "type": ["string", "null"] }, + "powo_id": { "type": ["string", "null"] } + } + } + }, + "$defs": { + "slug": { "type": "string", "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$" }, + "status": { "enum": ["stub", "enriched", "reviewed", "confirmed"] }, + "appearances": { + "type": "object", + "additionalProperties": false, + "properties": { + "mishnah": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)* \\d+:\\d+$" + } + }, + "other": { "type": "array", "items": { "type": "string" } } + } + } + } +} diff --git a/entities/schema/source.schema.json b/entities/schema/source.schema.json new file mode 100644 index 0000000..3533c13 --- /dev/null +++ b/entities/schema/source.schema.json @@ -0,0 +1,40 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "source.schema.json", + "title": "Source registry", + "description": "Schema for data/sources.yaml as a whole: a map of source key -> source entry. Provenance fields elsewhere reference these keys. See docs/entities-knowledge-base.md §6.", + "type": "object", + "minProperties": 1, + "propertyNames": { + "pattern": "^[a-z0-9]+(?:[-_][a-z0-9]+)*$" + }, + "additionalProperties": { + "$ref": "#/$defs/source" + }, + "$defs": { + "source": { + "type": "object", + "required": ["citation", "type", "license", "trust_tier"], + "additionalProperties": false, + "properties": { + "citation": { + "type": "string", + "minLength": 1 + }, + "type": { + "enum": ["structured", "scholarly-prose", "derived", "human"] + }, + "license": { + "type": "string", + "minLength": 1, + "description": "Free text (e.g. CC0, CC-BY, public-domain, in-copyright, project). Determines what may be redistributed." + }, + "trust_tier": { + "type": "integer", + "minimum": 0, + "description": "Lower number == higher trust. 0 == manual curation." + } + } + } + } +} diff --git a/scripts/kb-validate.py b/scripts/kb-validate.py new file mode 100644 index 0000000..b90b6ba --- /dev/null +++ b/scripts/kb-validate.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +"""Validate the entities knowledge base. + +Usage: + python3 scripts/kb-validate.py + python3 scripts/kb-validate.py --data entities/data --schema entities/schema + +Validation is the contract: a file that does not validate does not land +(wire this into a pre-commit hook and CI). See docs/entities-knowledge-base.md §11. + +Two layers of checks: + + 1. JSON Schema — every entities/data/**/*.yaml against its schema in + entities/schema/ (people/ -> person, places/ -> place, plants/ -> plant), + and sources.yaml against source.schema.json. + + 2. Semantic cross-checks the schemas can't express: + - every claim's `source` exists in data/sources.yaml + - every mishnah appearance references a canonical masechet slug + (from data/vocab/sedarim.yaml) + - each entity file's stem matches its `slug` + + Cross-entity referential integrity (relationship/locale/alias slugs resolving + to real files) is intentionally NOT enforced yet — entities accumulate over + time and may reference not-yet-created stubs. + +Exits non-zero if anything fails. + +Requires: pyyaml, jsonschema (>=4.18). +""" + +import argparse +import json +import sys +from pathlib import Path + +try: + import yaml +except ImportError: + sys.exit("kb-validate: PyYAML is required — pip install pyyaml") + +try: + from jsonschema import Draft202012Validator + from referencing import Registry, Resource +except ImportError: + sys.exit("kb-validate: jsonschema>=4.18 is required — pip install jsonschema") + +ROOT = Path(__file__).resolve().parent.parent +FOLDER_SCHEMA = {"people": "person", "places": "place", "plants": "plant"} + +# ANSI, but degrade gracefully when not a tty. +_TTY = sys.stdout.isatty() +GREEN = "\033[32m" if _TTY else "" +RED = "\033[31m" if _TTY else "" +DIM = "\033[2m" if _TTY else "" +RESET = "\033[0m" if _TTY else "" +OK = f"{GREEN}✓{RESET}" +BAD = f"{RED}✗{RESET}" + + +def load_yaml(path): + with open(path, encoding="utf-8") as f: + return yaml.safe_load(f) + + +def build_registry(schema_dir): + """Load every *.schema.json into a registry keyed by its $id so that + cross-file $refs (e.g. "claim.schema.json") resolve.""" + registry = Registry() + schemas = {} + for path in sorted(schema_dir.glob("*.schema.json")): + schema = json.loads(path.read_text(encoding="utf-8")) + uri = schema.get("$id", path.name) + registry = registry.with_resource(uri, Resource.from_contents(schema)) + schemas[path.name] = schema + return registry, schemas + + +def iter_claims(node): + """Yield every claim-shaped dict (any mapping carrying a `source` key) + anywhere in the entity tree.""" + if isinstance(node, dict): + if "source" in node and isinstance(node.get("source"), str): + yield node + for v in node.values(): + yield from iter_claims(v) + elif isinstance(node, list): + for v in node: + yield from iter_claims(v) + + +def masechet_slugs(data_dir): + sedarim = load_yaml(data_dir / "vocab" / "sedarim.yaml") or {} + return {m["slug"] for masechtot in sedarim.values() for m in masechtot} + + +def schema_errors(instance, schema, registry): + validator = Draft202012Validator(schema, registry=registry) + out = [] + for err in sorted(validator.iter_errors(instance), key=lambda e: list(e.path)): + loc = "/".join(str(p) for p in err.path) or "(root)" + out.append(f"{loc}: {err.message}") + return out + + +def main(): + ap = argparse.ArgumentParser(description="Validate the entities knowledge base.") + ap.add_argument("--data", default=str(ROOT / "entities" / "data")) + ap.add_argument("--schema", default=str(ROOT / "entities" / "schema")) + args = ap.parse_args() + + data_dir = Path(args.data) + schema_dir = Path(args.schema) + + if not data_dir.is_dir() or not schema_dir.is_dir(): + sys.exit(f"kb-validate: missing data/ or schema/ dir ({data_dir}, {schema_dir})") + + registry, schemas = build_registry(schema_dir) + sources = load_yaml(data_dir / "sources.yaml") or {} + source_keys = set(sources) + slugs = masechet_slugs(data_dir) + + files_ok = 0 + files_bad = 0 + + def report(path, errors): + nonlocal files_ok, files_bad + try: + rel = path.relative_to(ROOT) + except ValueError: + rel = path + if errors: + files_bad += 1 + print(f"{BAD} {rel}") + for e in errors: + print(f" {DIM}- {e}{RESET}") + else: + files_ok += 1 + print(f"{OK} {rel}") + + # 1. sources.yaml against source.schema.json + report(data_dir / "sources.yaml", + schema_errors(sources, schemas["source.schema.json"], registry)) + + # 2. each entity file + for folder, kind in FOLDER_SCHEMA.items(): + sub = data_dir / folder + if not sub.is_dir(): + continue + schema = schemas[f"{kind}.schema.json"] + for path in sorted(sub.glob("*.yaml")): + doc = load_yaml(path) + errors = schema_errors(doc, schema, registry) + + # semantic checks (only meaningful once the doc is a mapping) + if isinstance(doc, dict): + if doc.get("slug") != path.stem: + errors.append( + f"slug: '{doc.get('slug')}' does not match filename '{path.stem}'") + for claim in iter_claims(doc): + src = claim.get("source") + if src not in source_keys: + errors.append(f"source '{src}' is not in sources.yaml") + refs = (doc.get("appearances") or {}).get("mishnah") or [] + for ref in refs: + slug = ref.rsplit(" ", 1)[0] + if slug not in slugs: + errors.append( + f"appearance '{ref}': '{slug}' is not a canonical masechet slug") + + report(path, errors) + + total = files_ok + files_bad + print() + if files_bad: + print(f"{BAD} {files_bad}/{total} file(s) failed validation.") + return 1 + print(f"{OK} all {total} file(s) valid.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 58371cf41bc3afa96ac7b3c79e31929bf900c02d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 10:08:07 +0000 Subject: [PATCH 4/7] Phase 1 (deliverable 6): kb-build.py compiles YAML into knowledge.db - Flattened tables: source, entity, claim, appearance, external_id (claim extras like candidate ids preserved in extra_json; mishnah refs parsed to masechet slug) - Derived tables built at compile time: contemporaries, cooccurrence, region_rollup - Deterministic & idempotent: rebuilt from scratch, byte-identical across runs - knowledge.db stays gitignored (derived artifact) --- README.md | 2 +- scripts/kb-build.py | 225 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 226 insertions(+), 1 deletion(-) create mode 100644 scripts/kb-build.py diff --git a/README.md b/README.md index deb108d..200bb95 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ This repo contains: - `scripts/fix.py` — programmatic fixes + LLM regen for errors - `scripts/merge.py` — apply JSON corrections into HTML files - `scripts/update-readme.py` — regenerate masechot table from meta tags -- **Entities knowledge base** (`entities/`) — an accumulating store of people, places, and plants in the Mishnah ([KB spec](docs/entities-knowledge-base.md), [display spec](docs/entities-display.md)). `scripts/kb-validate.py` validates `entities/data/` against the JSON schemas (run `pip install -r entities/requirements.txt` first). +- **Entities knowledge base** (`entities/`) — an accumulating store of people, places, and plants in the Mishnah ([KB spec](docs/entities-knowledge-base.md), [display spec](docs/entities-display.md)). YAML is the source of truth; `scripts/kb-validate.py` validates it against the JSON schemas and `scripts/kb-build.py` compiles it into the derived `entities/knowledge.db` (run `pip install -r entities/requirements.txt` first). - **A Claude skill** (`.claude/skills/mishnah/`) for interactive formatting in Claude Code ## The Style diff --git a/scripts/kb-build.py b/scripts/kb-build.py new file mode 100644 index 0000000..f6b61e1 --- /dev/null +++ b/scripts/kb-build.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Compile the entities knowledge base YAML into knowledge.db (SQLite). + +Usage: + python3 scripts/kb-build.py + python3 scripts/kb-build.py --data entities/data --out entities/knowledge.db + +The DB is a derived, regenerable index — never hand-edited; rebuild from +entities/data/. The build is deterministic and idempotent: it drops and +recreates everything from scratch, so the same YAML in produces the same DB out. +See docs/entities-knowledge-base.md §11. + +Tables (flattened — they do not mirror the YAML nesting): + source(key, citation, type, license, trust_tier) + entity(slug, kind, type, status, he, en) + claim(entity_slug, field, value_json, source, confidence, + confirmed, asserted_by, asserted_date, note, extra_json) + appearance(entity_slug, ref, work, masechet) + external_id(entity_slug, authority, id) +Derived at build time: + contemporaries(slug_a, slug_b, generation) -- people sharing a generation + cooccurrence(slug_a, slug_b, ref) -- entities in the same mishna + region_rollup(region, place_count) + +Run scripts/kb-validate.py first; this build assumes valid input. + +Requires: pyyaml. +""" + +import argparse +import json +import sqlite3 +import sys +from itertools import combinations +from pathlib import Path + +try: + import yaml +except ImportError: + sys.exit("kb-build: PyYAML is required — pip install pyyaml") + +ROOT = Path(__file__).resolve().parent.parent +KINDS = {"people": "person", "places": "place", "plants": "plant"} +PROVENANCE_KEYS = {"value", "source", "confidence", "asserted_by", "date", "confirmed", "note"} + +SCHEMA = """ +CREATE TABLE source ( + key TEXT PRIMARY KEY, citation TEXT, type TEXT, license TEXT, trust_tier INTEGER +); +CREATE TABLE entity ( + slug TEXT PRIMARY KEY, kind TEXT, type TEXT, status TEXT, he TEXT, en TEXT +); +CREATE TABLE claim ( + entity_slug TEXT, field TEXT, value_json TEXT, source TEXT, confidence TEXT, + confirmed INTEGER, asserted_by TEXT, asserted_date TEXT, note TEXT, extra_json TEXT +); +CREATE TABLE appearance ( + entity_slug TEXT, ref TEXT, work TEXT, masechet TEXT +); +CREATE TABLE external_id ( + entity_slug TEXT, authority TEXT, id TEXT +); +CREATE TABLE contemporaries (slug_a TEXT, slug_b TEXT, generation INTEGER); +CREATE TABLE cooccurrence (slug_a TEXT, slug_b TEXT, ref TEXT); +CREATE TABLE region_rollup (region TEXT, place_count INTEGER); +""" + + +def load_yaml(path): + with open(path, encoding="utf-8") as f: + return yaml.safe_load(f) + + +def jdump(obj): + return json.dumps(obj, ensure_ascii=False, sort_keys=True) + + +def walk_claims(node, path=()): + """Yield (field, claim) for every claim-shaped mapping (carrying a string + `source`). `field` is the dotted path to the key holding the claim list, + e.g. 'bio', 'geo.coordinates', 'identification.candidates'.""" + if isinstance(node, dict): + if isinstance(node.get("source"), str): + yield ".".join(path), node + return # a claim is a leaf — don't descend into its `value` + for key, val in node.items(): + yield from walk_claims(val, path + (key,)) + elif isinstance(node, list): + for item in node: + yield from walk_claims(item, path) + + +def he_en(kind, doc): + if kind == "plant": + term = doc.get("term") or {} + return term.get("he"), term.get("en_common") + names = doc.get("names") or {} + return names.get("he"), names.get("en") + + +def entity_type(kind, doc): + return doc.get("term_type") if kind == "plant" else doc.get("type") + + +def load_entities(data_dir): + """Return a list of (kind, slug, doc) sorted by (kind, slug).""" + out = [] + for folder, kind in KINDS.items(): + sub = data_dir / folder + if not sub.is_dir(): + continue + for path in sorted(sub.glob("*.yaml")): + out.append((kind, path.stem, load_yaml(path))) + return sorted(out, key=lambda t: (t[0], t[1])) + + +def build(data_dir, out_path): + out_path = Path(out_path) + if out_path.exists(): + out_path.unlink() + out_path.parent.mkdir(parents=True, exist_ok=True) + + db = sqlite3.connect(out_path) + db.executescript(SCHEMA) + + # sources + sources = load_yaml(data_dir / "sources.yaml") or {} + for key in sorted(sources): + s = sources[key] + db.execute( + "INSERT INTO source VALUES (?,?,?,?,?)", + (key, s.get("citation"), s.get("type"), s.get("license"), s.get("trust_tier")), + ) + + entities = load_entities(data_dir) + # ref -> set of slugs, for co-occurrence; generation -> slugs, for contemporaries + ref_entities, gen_people, region_counts = {}, {}, {} + + for kind, slug, doc in entities: + he, en = he_en(kind, doc) + db.execute( + "INSERT INTO entity VALUES (?,?,?,?,?,?)", + (slug, kind, entity_type(kind, doc), doc.get("status"), he, en), + ) + + for field, claim in walk_claims(doc): + extra = {k: v for k, v in claim.items() if k not in PROVENANCE_KEYS} + db.execute( + "INSERT INTO claim VALUES (?,?,?,?,?,?,?,?,?,?)", + ( + slug, field, jdump(claim.get("value")), claim.get("source"), + claim.get("confidence"), 1 if claim.get("confirmed") else 0, + claim.get("asserted_by"), claim.get("date"), claim.get("note"), + jdump(extra) if extra else None, + ), + ) + + appearances = doc.get("appearances") or {} + for work in ("mishnah", "other"): + for ref in appearances.get(work) or []: + masechet = ref.rsplit(" ", 1)[0] if work == "mishnah" else None + db.execute( + "INSERT INTO appearance VALUES (?,?,?,?)", (slug, ref, work, masechet) + ) + if work == "mishnah": + ref_entities.setdefault(ref, set()).add(slug) + + for authority, value in sorted((doc.get("ids") or {}).items()): + if value is not None: + db.execute( + "INSERT INTO external_id VALUES (?,?,?)", (slug, authority, str(value)) + ) + + if kind == "person": + gen = (doc.get("era") or {}).get("generation") + if gen is not None: + gen_people.setdefault(gen, []).append(slug) + if kind == "place": + region = (doc.get("geo") or {}).get("region") if doc.get("geo") else None + if region: + region_counts[region] = region_counts.get(region, 0) + 1 + + # derived: contemporaries (same tannaitic generation) + for gen in sorted(gen_people): + for a, b in combinations(sorted(gen_people[gen]), 2): + db.execute("INSERT INTO contemporaries VALUES (?,?,?)", (a, b, gen)) + + # derived: co-occurrence (entities sharing a mishna ref) + for ref in sorted(ref_entities): + for a, b in combinations(sorted(ref_entities[ref]), 2): + db.execute("INSERT INTO cooccurrence VALUES (?,?,?)", (a, b, ref)) + + # derived: region rollup + for region in sorted(region_counts): + db.execute("INSERT INTO region_rollup VALUES (?,?)", (region, region_counts[region])) + + db.commit() + counts = { + t: db.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0] + for t in ("source", "entity", "claim", "appearance", "external_id", + "contemporaries", "cooccurrence", "region_rollup") + } + db.close() + return counts + + +def main(): + ap = argparse.ArgumentParser(description="Compile the entities KB into knowledge.db.") + ap.add_argument("--data", default=str(ROOT / "entities" / "data")) + ap.add_argument("--out", default=str(ROOT / "entities" / "knowledge.db")) + args = ap.parse_args() + + data_dir = Path(args.data) + if not data_dir.is_dir(): + sys.exit(f"kb-build: data dir not found: {data_dir}") + + counts = build(data_dir, args.out) + print(f"built {args.out}") + for table, n in counts.items(): + print(f" {table:<15} {n}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From e26405b2e43db09af4ebadf7fbb9ad5baed37f4c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 10:17:45 +0000 Subject: [PATCH 5/7] =?UTF-8?q?Phase=201=20(deliverable=207):=20importers?= =?UTF-8?q?=20+=20=C2=A78=20upsert=20engine?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scripts/kb_lib.py: comment-preserving (ruamel) round-trip + §8 merge rules (additive/idempotent upserts, confirmed-protection -> conflicts.log, appearance dedupe, single vs multi-valued claim modes) - scripts/kb-import-wikidata.py: enrich people/places/plants (plants via --qid, since the taxon QID lives on the candidate); fetch or --input - scripts/kb-import-pleiades.py: place coordinate claims; fetch or --input - scripts/kb-selftest.py: offline assertions of the merge-rule invariants - entities/fixtures/: API-shaped JSON so the full pipeline runs offline - CI now runs validate + selftest + build; requirements add ruamel.yaml --- .github/workflows/kb-validate.yml | 6 +- README.md | 6 +- entities/fixtures/pleiades/678378.json | 6 + entities/fixtures/wikidata/Q12100.json | 24 +++ entities/fixtures/wikidata/Q310357.json | 19 +++ entities/fixtures/wikidata/Q745966.json | 32 ++++ entities/requirements.txt | 5 +- scripts/kb-import-pleiades.py | 100 ++++++++++++ scripts/kb-import-wikidata.py | 173 +++++++++++++++++++++ scripts/kb-selftest.py | 76 +++++++++ scripts/kb_lib.py | 195 ++++++++++++++++++++++++ 11 files changed, 637 insertions(+), 5 deletions(-) create mode 100644 entities/fixtures/pleiades/678378.json create mode 100644 entities/fixtures/wikidata/Q12100.json create mode 100644 entities/fixtures/wikidata/Q310357.json create mode 100644 entities/fixtures/wikidata/Q745966.json create mode 100644 scripts/kb-import-pleiades.py create mode 100644 scripts/kb-import-wikidata.py create mode 100644 scripts/kb-selftest.py create mode 100644 scripts/kb_lib.py diff --git a/.github/workflows/kb-validate.yml b/.github/workflows/kb-validate.yml index 2eeb872..ab15322 100644 --- a/.github/workflows/kb-validate.yml +++ b/.github/workflows/kb-validate.yml @@ -4,12 +4,12 @@ on: push: paths: - "entities/**" - - "scripts/kb-validate.py" + - "scripts/kb-*.py" - ".github/workflows/kb-validate.yml" pull_request: paths: - "entities/**" - - "scripts/kb-validate.py" + - "scripts/kb-*.py" - ".github/workflows/kb-validate.yml" jobs: @@ -22,3 +22,5 @@ jobs: python-version: "3.11" - run: pip install -r entities/requirements.txt - run: python3 scripts/kb-validate.py + - run: python3 scripts/kb-selftest.py # §8 merge-rule invariants (offline) + - run: python3 scripts/kb-build.py # smoke: YAML compiles to knowledge.db diff --git a/README.md b/README.md index 200bb95..423aaeb 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,11 @@ This repo contains: - `scripts/fix.py` — programmatic fixes + LLM regen for errors - `scripts/merge.py` — apply JSON corrections into HTML files - `scripts/update-readme.py` — regenerate masechot table from meta tags -- **Entities knowledge base** (`entities/`) — an accumulating store of people, places, and plants in the Mishnah ([KB spec](docs/entities-knowledge-base.md), [display spec](docs/entities-display.md)). YAML is the source of truth; `scripts/kb-validate.py` validates it against the JSON schemas and `scripts/kb-build.py` compiles it into the derived `entities/knowledge.db` (run `pip install -r entities/requirements.txt` first). +- **Entities knowledge base** (`entities/`) — an accumulating store of people, places, and plants in the Mishnah ([KB spec](docs/entities-knowledge-base.md), [display spec](docs/entities-display.md)). YAML is the source of truth (run `pip install -r entities/requirements.txt` first): + - `scripts/kb-validate.py` — validate `entities/data/` against the JSON schemas + semantic cross-checks + - `scripts/kb-build.py` — compile the YAML into the derived `entities/knowledge.db` + - `scripts/kb-import-wikidata.py`, `scripts/kb-import-pleiades.py` — enrich entities from external sources (additive, idempotent; `--input` runs offline against `entities/fixtures/`) + - `scripts/kb-selftest.py` — assert the §8 merge-rule invariants offline - **A Claude skill** (`.claude/skills/mishnah/`) for interactive formatting in Claude Code ## The Style diff --git a/entities/fixtures/pleiades/678378.json b/entities/fixtures/pleiades/678378.json new file mode 100644 index 0000000..20decd3 --- /dev/null +++ b/entities/fixtures/pleiades/678378.json @@ -0,0 +1,6 @@ +{ + "id": "678378", + "title": "Sepphoris/Diocaesarea", + "reprPoint": [35.2797, 32.7522], + "placeTypes": ["settlement"] +} diff --git a/entities/fixtures/wikidata/Q12100.json b/entities/fixtures/wikidata/Q12100.json new file mode 100644 index 0000000..7c6ab30 --- /dev/null +++ b/entities/fixtures/wikidata/Q12100.json @@ -0,0 +1,24 @@ +{ + "entities": { + "Q12100": { + "id": "Q12100", + "labels": { "en": { "language": "en", "value": "common wheat" } }, + "claims": { + "P225": [ + { + "mainsnak": { + "datavalue": { "type": "string", "value": "Triticum aestivum" } + } + } + ], + "P105": [ + { + "mainsnak": { + "datavalue": { "type": "wikibase-entityid", "value": { "id": "Q7432" } } + } + } + ] + } + } + } +} diff --git a/entities/fixtures/wikidata/Q310357.json b/entities/fixtures/wikidata/Q310357.json new file mode 100644 index 0000000..4ae5478 --- /dev/null +++ b/entities/fixtures/wikidata/Q310357.json @@ -0,0 +1,19 @@ +{ + "entities": { + "Q310357": { + "id": "Q310357", + "labels": { + "en": { "language": "en", "value": "Akiva" }, + "he": { "language": "he", "value": "רבי עקיבא" } + }, + "descriptions": { + "en": { "language": "en", "value": "Jewish scholar and tanna (c. 50 – 135 CE)" } + }, + "aliases": { + "en": [ { "value": "Akiva ben Joseph" }, { "value": "Rabbi Akiba" } ], + "he": [ { "value": "עקיבא בן יוסף" } ] + }, + "claims": {} + } + } +} diff --git a/entities/fixtures/wikidata/Q745966.json b/entities/fixtures/wikidata/Q745966.json new file mode 100644 index 0000000..10a043e --- /dev/null +++ b/entities/fixtures/wikidata/Q745966.json @@ -0,0 +1,32 @@ +{ + "entities": { + "Q745966": { + "id": "Q745966", + "labels": { "en": { "language": "en", "value": "Sepphoris" } }, + "claims": { + "P625": [ + { + "mainsnak": { + "datavalue": { + "type": "globecoordinate", + "value": { + "latitude": 32.7522, + "longitude": 35.2797, + "precision": 0.0001, + "globe": "http://www.wikidata.org/entity/Q2" + } + } + } + } + ], + "P18": [ + { + "mainsnak": { + "datavalue": { "type": "string", "value": "Sepphoris Cardo.jpg" } + } + } + ] + } + } + } +} diff --git a/entities/requirements.txt b/entities/requirements.txt index 977c4e2..53b175d 100644 --- a/entities/requirements.txt +++ b/entities/requirements.txt @@ -1,3 +1,4 @@ # Dependencies for the entities knowledge base tooling (scripts/kb-*.py). -PyYAML>=6.0 -jsonschema>=4.18 +PyYAML>=6.0 # kb-validate, kb-build +jsonschema>=4.18 # kb-validate +ruamel.yaml>=0.18 # kb_lib, kb-import-*, kb-selftest (comment-preserving round-trip) diff --git a/scripts/kb-import-pleiades.py b/scripts/kb-import-pleiades.py new file mode 100644 index 0000000..813c7f6 --- /dev/null +++ b/scripts/kb-import-pleiades.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""Import place coordinates from Pleiades. + +Usage: + python3 scripts/kb-import-pleiades.py --slug tzippori + python3 scripts/kb-import-pleiades.py --all --dry-run + python3 scripts/kb-import-pleiades.py --slug tzippori --input fixture.json + +Enriches places that carry an `ids.pleiades_id`, appending a coordinate claim +(source: pleiades) through the §8 upsert rules. Coordinate claims are +multi-valued — the Pleiades point is retained alongside any others as dissent. + +Network note: the live fetch hits pleiades.stoa.org and needs a network policy +that allows it. Use `--input PATH` to run against saved JSON (offline). + +Requires: ruamel.yaml (and network access, unless --input is used). +""" + +import argparse +import json +import sys +import urllib.request +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import kb_lib as kb + +ROOT = Path(__file__).resolve().parent.parent +USER_AGENT = "mishnah-style-kb/0.1 (https://github.com/mig2/mishnah-style)" +PLEIADES_URL = "https://pleiades.stoa.org/places/{pid}/json" + + +def fetch(pid): + req = urllib.request.Request(PLEIADES_URL.format(pid=pid), + headers={"User-Agent": USER_AGENT}) + with urllib.request.urlopen(req, timeout=30) as resp: + return json.load(resp) + + +def transform(pl, data, conflicts): + """Pleiades reprPoint is [lon, lat] (GeoJSON order).""" + pt = pl.get("reprPoint") + if not pt or len(pt) != 2: + return [] + coord = kb.flow_map({"lat": pt[1], "lon": pt[0]}) + status = kb.upsert_claim( + data, "geo.coordinates", coord, "pleiades", mode="multi", + conflicts=conflicts, entity_slug=data.get("slug", ""), + confidence="accepted", date=kb.today()) + return [("geo.coordinates", status)] + + +def main(): + ap = argparse.ArgumentParser(description="Enrich place coordinates from Pleiades.") + ap.add_argument("--slug") + ap.add_argument("--all", action="store_true") + ap.add_argument("--input", help="Read Pleiades JSON from a file instead of fetching.") + ap.add_argument("--dry-run", action="store_true") + ap.add_argument("--data", default=str(ROOT / "entities" / "data")) + args = ap.parse_args() + + sub = Path(args.data) / "places" + if args.slug: + paths = [sub / f"{args.slug}.yaml"] + elif args.all: + paths = sorted(sub.glob("*.yaml")) + else: + sys.exit("kb-import-pleiades: pass --slug or --all") + + conflicts = [] + changed = 0 + for path in paths: + if not path.exists(): + print(f" skip {path.name}: no such place") + continue + data = kb.load_entity(path) + if data.get("geo") is None: + print(f" skip {path.name}: no geo block") + continue + pid = (data.get("ids") or {}).get("pleiades_id") + if not pid: + print(f" skip {path.name}: no pleiades_id") + continue + pl = json.loads(Path(args.input).read_text()) if args.input else fetch(pid) + log = transform(pl, data, conflicts) + touched = [f"{f}={s}" for f, s in log if s != "unchanged"] + if touched: + changed += 1 + print(f" {path.name} ({pid}): {', '.join(touched)}") + if not args.dry_run: + kb.save_entity(path, data) + else: + print(f" {path.name} ({pid}): no change") + + print(f"\n{'(dry-run) ' if args.dry_run else ''}{changed} place(s) changed.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/kb-import-wikidata.py b/scripts/kb-import-wikidata.py new file mode 100644 index 0000000..6e32544 --- /dev/null +++ b/scripts/kb-import-wikidata.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Import / enrich entities from Wikidata. + +Usage: + python3 scripts/kb-import-wikidata.py --kind people --slug akiva + python3 scripts/kb-import-wikidata.py --kind places --all --dry-run + python3 scripts/kb-import-wikidata.py --kind plants --slug chitah --input fixture.json + +One writer among several (docs/entities-knowledge-base.md §10): it enriches +entities that already carry an `ids.wikidata_qid`, appending claims through the +§8 upsert rules. It never creates stubs and never overwrites confirmed claims. + +Network note: the live fetch hits the Wikidata REST endpoint and needs a network +policy that allows wikidata.org. Use `--input PATH` to run the same transform +against a saved EntityData JSON (offline, e.g. the fixtures in entities/fixtures/). + +Requires: ruamel.yaml (and network access, unless --input is used). +""" + +import argparse +import json +import sys +import urllib.request +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import kb_lib as kb + +ROOT = Path(__file__).resolve().parent.parent +USER_AGENT = "mishnah-style-kb/0.1 (https://github.com/mig2/mishnah-style)" +WIKIDATA_URL = "https://www.wikidata.org/wiki/Special:EntityData/{qid}.json" + +# Wikidata taxon-rank items (P105) -> rank label +RANKS = {"Q7432": "species", "Q34740": "genus", "Q35409": "family", + "Q37517": "subspecies", "Q38829": "variety"} + + +def fetch(qid): + req = urllib.request.Request(WIKIDATA_URL.format(qid=qid), + headers={"User-Agent": USER_AGENT}) + with urllib.request.urlopen(req, timeout=30) as resp: + return json.load(resp) + + +def _entity_blob(wd, qid): + return (wd.get("entities") or {}).get(qid) or {} + + +def _label(blob, lang): + return ((blob.get("labels") or {}).get(lang) or {}).get("value") + + +def _aliases(blob, lang): + return [a.get("value") for a in (blob.get("aliases") or {}).get(lang, []) if a.get("value")] + + +def _claim_values(blob, prop): + out = [] + for st in (blob.get("claims") or {}).get(prop, []): + dv = ((st.get("mainsnak") or {}).get("datavalue") or {}).get("value") + if dv is not None: + out.append(dv) + return out + + +def transform(kind, qid, wd, data, conflicts): + """Apply Wikidata facts to `data`. Returns a list of (field, status).""" + blob = _entity_blob(wd, qid) + log = [] + d = kb.today() + + if kind == "person": + desc = ((blob.get("descriptions") or {}).get("en") or {}).get("value") + if desc: + log.append(("bio", kb.upsert_claim( + data, "bio", desc, "wikidata", mode="single", conflicts=conflicts, + entity_slug=data.get("slug", ""), confidence="accepted", date=d))) + variants = _aliases(blob, "he") + _aliases(blob, "en") + if variants: + n = kb.merge_variants(data, "names.variants", variants) + log.append(("names.variants", f"+{n}" if n else "unchanged")) + + elif kind == "place": + if data.get("geo") is not None: + for v in _claim_values(blob, "P625"): + coord = kb.flow_map({"lat": v.get("latitude"), "lon": v.get("longitude")}) + log.append(("geo.coordinates", kb.upsert_claim( + data, "geo.coordinates", coord, "wikidata", mode="multi", + conflicts=conflicts, entity_slug=data.get("slug", ""), + confidence="probable", date=d))) + photos = _claim_values(blob, "P18") + if photos and not (data.get("media") or {}).get("photo"): + url = "https://commons.wikimedia.org/wiki/File:" + str(photos[0]).replace(" ", "_") + data.setdefault("media", kb.CommentedMap())["photo"] = url + log.append(("media.photo", "set")) + + elif kind == "plant": + taxa = _claim_values(blob, "P225") + rank_qids = _claim_values(blob, "P105") + rank = RANKS.get((rank_qids[0] or {}).get("id")) if rank_qids else None + for taxon in taxa: + value = kb.flow_map({"taxon": taxon, "rank": rank or "species"}) + extra = {"ids": kb.flow_map({"wikidata_qid": qid})} + log.append(("identification.candidates", kb.upsert_claim( + data, "identification.candidates", value, "wikidata", mode="multi", + conflicts=conflicts, entity_slug=data.get("slug", ""), + confidence="probable", date=d, extra=extra))) + + return log + + +def targets(data_dir, kind, slug, want_all): + folder = kb.KIND_DIR.get(kind, kind) + sub = Path(data_dir) / folder + if slug: + return [sub / f"{slug}.yaml"] + if want_all: + return sorted(sub.glob("*.yaml")) + sys.exit("kb-import-wikidata: pass --slug or --all") + + +def main(): + ap = argparse.ArgumentParser(description="Enrich entities from Wikidata.") + ap.add_argument("--kind", required=True, choices=["people", "places", "plants"]) + ap.add_argument("--slug") + ap.add_argument("--all", action="store_true") + ap.add_argument("--qid", help="Taxon/entity QID to import. Required for plants " + "(the QID lives on the candidate, not the term); " + "overrides ids.wikidata_qid for people/places.") + ap.add_argument("--input", help="Read EntityData JSON from a file instead of fetching.") + ap.add_argument("--dry-run", action="store_true") + ap.add_argument("--data", default=str(ROOT / "entities" / "data")) + args = ap.parse_args() + + kind = kb.DIR_KIND[args.kind] + conflicts = [] + changed = 0 + + for path in targets(args.data, args.kind, args.slug, args.all): + if not path.exists(): + print(f" skip {path.name}: no such entity") + continue + data = kb.load_entity(path) + qid = args.qid or (data.get("ids") or {}).get("wikidata_qid") + if not qid: + hint = " (plants need --qid)" if kind == "plant" else "" + print(f" skip {path.name}: no wikidata_qid{hint}") + continue + wd = json.loads(Path(args.input).read_text()) if args.input else fetch(qid) + log = transform(kind, qid, wd, data, conflicts) + touched = [f"{field}={status}" for field, status in log + if status not in ("unchanged", "0")] + if touched: + changed += 1 + print(f" {path.name} ({qid}): {', '.join(touched)}") + if not args.dry_run: + kb.save_entity(path, data) + else: + print(f" {path.name} ({qid}): no change") + + if conflicts: + print(f"\n⚠ {len(conflicts)} conflict(s) with confirmed claims — NOT written:") + for c in conflicts: + print(f" {c['entity']}.{c['field']}: kept {c['kept']!r}, rejected {c['rejected']!r}") + if not args.dry_run: + kb.write_conflicts(conflicts, ROOT / "entities" / "conflicts.log") + + print(f"\n{'(dry-run) ' if args.dry_run else ''}{changed} entit(ies) changed.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/kb-selftest.py b/scripts/kb-selftest.py new file mode 100644 index 0000000..0f3ab2c --- /dev/null +++ b/scripts/kb-selftest.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Offline self-test for the §8 merge rules in kb_lib. No network, no fixtures. +Run in CI to guarantee the upsert engine keeps its invariants. + +Requires: ruamel.yaml. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import kb_lib as kb +from ruamel.yaml.comments import CommentedMap + + +def entity(slug="x"): + d = CommentedMap() + d["slug"] = slug + return d + + +_failed = [] + + +def check(name, cond): + print(("\033[32m✓\033[0m " if cond else "\033[31m✗\033[0m ") + name) + if not cond: + _failed.append(name) + + +# 1. single: re-adding the same value is a no-op (idempotent, no churn) +d, conf = entity(), [] +kb.upsert_claim(d, "bio", "A", "wikidata", mode="single", conflicts=conf) +r = kb.upsert_claim(d, "bio", "A", "wikidata", mode="single", conflicts=conf) +check("single: re-add same value -> unchanged, no duplicate", + r == "unchanged" and len(d["bio"]) == 1) + +# 2. single: a changed value updates in place (one claim per source) +r = kb.upsert_claim(d, "bio", "B", "wikidata", mode="single", conflicts=conf) +check("single: changed value -> updated in place", + r == "updated" and len(d["bio"]) == 1 and d["bio"][0]["value"] == "B") + +# 3. single: a confirmed claim is never auto-overwritten; conflict is recorded +d2, conf2 = entity(), [] +kb.upsert_claim(d2, "bio", "keep", "manual", mode="single", conflicts=conf2, confirmed=True) +r = kb.upsert_claim(d2, "bio", "overwrite", "manual", mode="single", conflicts=conf2) +check("single: confirmed claim protected -> conflict, value kept, logged", + r == "conflict" and d2["bio"][0]["value"] == "keep" and len(conf2) == 1) + +# 4. multi: dedup on (value, source); a new value from any source is retained dissent +d3, conf3 = entity(), [] +kb.upsert_claim(d3, "geo.coordinates", {"lat": 1, "lon": 2}, "pleiades", mode="multi", conflicts=conf3) +r1 = kb.upsert_claim(d3, "geo.coordinates", {"lat": 1, "lon": 2}, "pleiades", mode="multi", conflicts=conf3) +r2 = kb.upsert_claim(d3, "geo.coordinates", {"lat": 9, "lon": 9}, "wikidata", mode="multi", conflicts=conf3) +check("multi: identical (value, source) -> unchanged", r1 == "unchanged") +check("multi: new value -> appended (dissent retained)", + r2 == "added" and len(d3["geo"]["coordinates"]) == 2) + +# 5. appearances dedupe by ref +d4 = entity() +a1 = kb.add_appearance(d4, "mishnah", "shekalim 4:2") +a2 = kb.add_appearance(d4, "mishnah", "shekalim 4:2") +check("appearance: deduped by ref", + a1 is True and a2 is False and len(d4["appearances"]["mishnah"]) == 1) + +# 6. external ids are idempotent +d5 = entity() +s1 = kb.set_external_id(d5, "wikidata_qid", "Q1") +s2 = kb.set_external_id(d5, "wikidata_qid", "Q1") +check("external_id: idempotent", s1 is True and s2 is False) + +print() +if _failed: + print(f"\033[31m✗ {len(_failed)} check(s) failed.\033[0m") + sys.exit(1) +print("\033[32m✓ all checks passed.\033[0m") diff --git a/scripts/kb_lib.py b/scripts/kb_lib.py new file mode 100644 index 0000000..f2b8246 --- /dev/null +++ b/scripts/kb_lib.py @@ -0,0 +1,195 @@ +"""Shared library for the entities knowledge base: round-trip YAML I/O and the +§8 merge rules. Importers (kb-import-*.py) are thin wrappers — fetch, transform, +then call these upserts. + +Every write obeys docs/entities-knowledge-base.md §8: + - additive, idempotent upserts: re-running an import produces zero churn + - NEVER auto-overwrite a `confirmed: true` claim — conflicts go to a review + queue (entities/conflicts.log), never the field + - appearances dedupe by ref + - single-valued fields: at most one claim per source (updatable in place) + - multi-valued fields (coordinates, candidates): append per (value, source) + +Round-trip YAML (ruamel) preserves comments and adjudication notes on write. + +Requires: ruamel.yaml. +""" + +from datetime import date as _date +from pathlib import Path + +from ruamel.yaml import YAML +from ruamel.yaml.comments import CommentedMap, CommentedSeq + +KIND_DIR = {"person": "people", "place": "places", "plant": "plants"} +DIR_KIND = {v: k for k, v in KIND_DIR.items()} +# canonical key order when a claim is first written +_CLAIM_ORDER = ["value", "source", "confidence", "asserted_by", "date", "confirmed", "note"] + + +def make_yaml(): + y = YAML() + y.preserve_quotes = True + y.width = 4096 # don't wrap long Hebrew/URLs + y.indent(mapping=2, sequence=2, offset=0) + return y + + +def today(): + return _date.today().isoformat() + + +def load_entity(path): + with open(path, encoding="utf-8") as f: + return make_yaml().load(f) + + +def save_entity(path, data): + with open(path, "w", encoding="utf-8") as f: + make_yaml().dump(data, f) + + +def entity_path(data_dir, kind, slug): + """kind may be singular ('person') or a folder ('people').""" + folder = KIND_DIR.get(kind, kind) + return Path(data_dir) / folder / f"{slug}.yaml" + + +def flow_map(d): + """A small mapping rendered inline (`{a: 1, b: 2}`) to match the hand-written + style of coordinate and taxon values.""" + m = CommentedMap(d) + m.fa.set_flow_style() + return m + + +def _plain(x): + """Strip ruamel wrappers so values compare by content.""" + if isinstance(x, dict): + return {k: _plain(v) for k, v in x.items()} + if isinstance(x, list): + return [_plain(v) for v in x] + return x + + +def _ensure_path_list(data, field): + """Navigate a dotted path, creating intermediate maps, and return the list + at the leaf (creating it if absent).""" + parts = field.split(".") + node = data + for p in parts[:-1]: + if node.get(p) is None: + node[p] = CommentedMap() + node = node[p] + last = parts[-1] + if node.get(last) is None: + node[last] = CommentedSeq() + return node[last] + + +def _make_claim(value, source, confidence, asserted_by, date, note, confirmed, extra): + c = CommentedMap() + c["value"] = value + c["source"] = source + if confidence is not None: + c["confidence"] = confidence + if asserted_by is not None: + c["asserted_by"] = asserted_by + if date is not None: + c["date"] = date + for k, v in (extra or {}).items(): + c[k] = v + c["confirmed"] = bool(confirmed) + if note is not None: + c["note"] = note + return c + + +def upsert_claim(data, field, value, source, *, mode, conflicts, entity_slug="", + confidence=None, asserted_by=None, date=None, note=None, + confirmed=False, extra=None): + """Upsert a claim into the list at `field`. Returns one of + 'unchanged' | 'added' | 'updated' | 'conflict'. + + mode='multi': append iff (value, source) is new; never overwrites. + mode='single': at most one claim per source. Same value -> no-op. + Different value -> update in place, UNLESS the existing claim is + confirmed, in which case record a conflict and leave it untouched. + """ + lst = _ensure_path_list(data, field) + pv = _plain(value) + + if mode == "multi": + for c in lst: + if c.get("source") == source and _plain(c.get("value")) == pv: + return "unchanged" + lst.append(_make_claim(value, source, confidence, asserted_by, date, + note, confirmed, extra)) + return "added" + + if mode == "single": + for c in lst: + if c.get("source") != source: + continue + if _plain(c.get("value")) == pv: + return "unchanged" + if c.get("confirmed"): + conflicts.append({ + "entity": entity_slug, "field": field, "source": source, + "kept": _plain(c.get("value")), "rejected": pv, + }) + return "conflict" + c["value"] = value + if confidence is not None: + c["confidence"] = confidence + if date is not None: + c["date"] = date + return "updated" + lst.append(_make_claim(value, source, confidence, asserted_by, date, + note, confirmed, extra)) + return "added" + + raise ValueError(f"unknown mode: {mode}") + + +def add_appearance(data, work, ref): + """Append a mishnah/other appearance ref, deduped. Returns True if added.""" + lst = _ensure_path_list(data, f"appearances.{work}") + if ref in lst: + return False + lst.append(ref) + return True + + +def set_external_id(data, authority, value): + """Set an external id (not a claim — no confirmed-protection). Idempotent.""" + if data.get("ids") is None: + data["ids"] = CommentedMap() + if data["ids"].get(authority) == value: + return False + data["ids"][authority] = value + return True + + +def merge_variants(data, field, values): + """Add string variants to the list at `field`, deduped, order-preserving. + Returns the count added.""" + lst = _ensure_path_list(data, field) + added = 0 + for v in values: + if v not in lst: + lst.append(v) + added += 1 + return added + + +def write_conflicts(conflicts, path): + """Append conflict records to the review queue (gitignored).""" + if not conflicts: + return + with open(path, "a", encoding="utf-8") as f: + for c in conflicts: + f.write( + f"{today()}\t{c['entity']}\t{c['field']}\tsource={c['source']}\t" + f"kept={c['kept']!r}\trejected={c['rejected']!r}\n" + ) From 1a41a7a2662f986ff076833df145f6bb5bdf26e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 11:03:10 +0000 Subject: [PATCH 6/7] Add complete test suite for entities KB (deliverables 1-7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 70 stdlib-unittest tests under tests/, one module per deliverable: D1-2 schemas (27) · D3 registries/vocab (9) · D4 validator (5) D5 exemplars (4) · D6 build (10) · D7 engine+importers (15) Each module is independently runnable; CI runs the full suite. --- .github/workflows/kb-validate.yml | 5 +- README.md | 3 +- tests/_util.py | 63 ++++++++++ tests/test_d1_d2_schemas.py | 159 +++++++++++++++++++++++++ tests/test_d3_registries.py | 56 +++++++++ tests/test_d4_validator.py | 63 ++++++++++ tests/test_d5_exemplars.py | 39 +++++++ tests/test_d6_build.py | 86 ++++++++++++++ tests/test_d7_importers.py | 185 ++++++++++++++++++++++++++++++ 9 files changed, 655 insertions(+), 4 deletions(-) create mode 100644 tests/_util.py create mode 100644 tests/test_d1_d2_schemas.py create mode 100644 tests/test_d3_registries.py create mode 100644 tests/test_d4_validator.py create mode 100644 tests/test_d5_exemplars.py create mode 100644 tests/test_d6_build.py create mode 100644 tests/test_d7_importers.py diff --git a/.github/workflows/kb-validate.yml b/.github/workflows/kb-validate.yml index ab15322..f816615 100644 --- a/.github/workflows/kb-validate.yml +++ b/.github/workflows/kb-validate.yml @@ -21,6 +21,5 @@ jobs: with: python-version: "3.11" - run: pip install -r entities/requirements.txt - - run: python3 scripts/kb-validate.py - - run: python3 scripts/kb-selftest.py # §8 merge-rule invariants (offline) - - run: python3 scripts/kb-build.py # smoke: YAML compiles to knowledge.db + - run: python3 scripts/kb-validate.py # gate: real data must validate + - run: python3 -m unittest discover -s tests -t tests -v # full suite (D1–D7) diff --git a/README.md b/README.md index 423aaeb..b078583 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,8 @@ This repo contains: - `scripts/kb-validate.py` — validate `entities/data/` against the JSON schemas + semantic cross-checks - `scripts/kb-build.py` — compile the YAML into the derived `entities/knowledge.db` - `scripts/kb-import-wikidata.py`, `scripts/kb-import-pleiades.py` — enrich entities from external sources (additive, idempotent; `--input` runs offline against `entities/fixtures/`) - - `scripts/kb-selftest.py` — assert the §8 merge-rule invariants offline + - `scripts/kb-selftest.py` — quick offline smoke of the §8 merge-rule invariants + - `tests/` — full suite, one module per deliverable: `python3 -m unittest discover -s tests -t tests` - **A Claude skill** (`.claude/skills/mishnah/`) for interactive formatting in Claude Code ## The Style diff --git a/tests/_util.py b/tests/_util.py new file mode 100644 index 0000000..e3fa2d7 --- /dev/null +++ b/tests/_util.py @@ -0,0 +1,63 @@ +"""Shared helpers for the entities-KB test suite (stdlib unittest, no new deps).""" + +import importlib.util +import json +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPTS = REPO / "scripts" +ENTITIES = REPO / "entities" +DATA = ENTITIES / "data" +SCHEMA = ENTITIES / "schema" +FIXTURES = ENTITIES / "fixtures" +MASECHOT = REPO / "masechot" + +# make `import kb_lib` work +sys.path.insert(0, str(SCRIPTS)) + + +def load_script(modname, filename): + """Import a (possibly hyphenated) script as a module.""" + spec = importlib.util.spec_from_file_location(modname, SCRIPTS / filename) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def load_yaml(path): + import yaml + with open(path, encoding="utf-8") as f: + return yaml.safe_load(f) + + +def load_registry(schema_dir=SCHEMA): + """Build a referencing registry from all schemas, keyed by $id.""" + from referencing import Registry, Resource + reg = Registry() + schemas = {} + for p in sorted(Path(schema_dir).glob("*.schema.json")): + s = json.loads(p.read_text(encoding="utf-8")) + schemas[p.name] = s + reg = reg.with_resource(s["$id"], Resource.from_contents(s)) + return reg, schemas + + +def validator_for(name): + """A Draft 2020-12 validator for the named schema (e.g. 'person', 'claim').""" + from jsonschema import Draft202012Validator + reg, schemas = load_registry() + return Draft202012Validator(schemas[f"{name}.schema.json"], registry=reg) + + +def is_valid(name, instance): + return not list(validator_for(name).iter_errors(instance)) + + +def masechot_slugs(): + return {p.stem for p in MASECHOT.glob("*.html") if p.stem != "index"} + + +def sedarim_slugs(): + data = load_yaml(DATA / "vocab" / "sedarim.yaml") + return {m["slug"] for masechtot in data.values() for m in masechtot} diff --git a/tests/test_d1_d2_schemas.py b/tests/test_d1_d2_schemas.py new file mode 100644 index 0000000..95d5370 --- /dev/null +++ b/tests/test_d1_d2_schemas.py @@ -0,0 +1,159 @@ +"""Deliverables 1–2: the five JSON schemas (claim, source, person, place, plant).""" + +import json +import unittest + +from _util import SCHEMA, load_registry, is_valid, validator_for + + +class TestSchemasWellFormed(unittest.TestCase): + def test_all_schema_files_are_valid_json_schema(self): + from jsonschema import Draft202012Validator + files = sorted(SCHEMA.glob("*.schema.json")) + self.assertEqual(len(files), 5, "expected 5 schema files") + for p in files: + with self.subTest(schema=p.name): + Draft202012Validator.check_schema(json.loads(p.read_text())) + + def test_cross_file_ref_resolves(self): + # person.bio items $ref "claim.schema.json"; building a validator must not raise + validator_for("person") + + +class TestClaim(unittest.TestCase): + def test_minimal_valid(self): + self.assertTrue(is_valid("claim", {"value": "x", "source": "wikidata"})) + + def test_source_required(self): + self.assertFalse(is_valid("claim", {"value": "x"})) + + def test_bad_confidence_rejected(self): + self.assertFalse(is_valid("claim", {"value": 1, "source": "s", "confidence": "maybe"})) + + def test_bad_date_rejected(self): + self.assertFalse(is_valid("claim", {"value": 1, "source": "s", "date": "2026/01/01"})) + + def test_extra_keys_allowed(self): + # candidates carry `ids`, medicinal carries `provenance` + self.assertTrue(is_valid("claim", {"value": 1, "source": "s", "ids": {"wikidata_qid": "Q1"}})) + + +class TestSource(unittest.TestCase): + def good(self): + return {"k": {"citation": "c", "type": "structured", "license": "CC0", "trust_tier": 1}} + + def test_valid(self): + self.assertTrue(is_valid("source", self.good())) + + def test_missing_required_rejected(self): + bad = self.good() + del bad["k"]["trust_tier"] + self.assertFalse(is_valid("source", bad)) + + def test_bad_type_enum_rejected(self): + bad = self.good() + bad["k"]["type"] = "blog" + self.assertFalse(is_valid("source", bad)) + + def test_negative_trust_tier_rejected(self): + bad = self.good() + bad["k"]["trust_tier"] = -1 + self.assertFalse(is_valid("source", bad)) + + +class TestPerson(unittest.TestCase): + def base(self): + return {"slug": "x", "status": "stub", "type": "tanna", "names": {"he": "פ"}} + + def test_minimal_valid(self): + self.assertTrue(is_valid("person", self.base())) + + def test_missing_names_rejected(self): + d = self.base() + del d["names"] + self.assertFalse(is_valid("person", d)) + + def test_bad_type_enum_rejected(self): + d = self.base() + d["type"] = "robot" + self.assertFalse(is_valid("person", d)) + + def test_additional_properties_rejected(self): + d = self.base() + d["surprise"] = 1 + self.assertFalse(is_valid("person", d)) + + def test_uppercase_slug_rejected(self): + d = self.base() + d["slug"] = "Rabbi_Akiva" + self.assertFalse(is_valid("person", d)) + + def test_generation_range_enforced(self): + d = self.base() + d["era"] = {"generation": 9} + self.assertFalse(is_valid("person", d)) + + def test_bio_claim_requires_source(self): + d = self.base() + d["bio"] = [{"value": "leading tanna"}] # no source + self.assertFalse(is_valid("person", d)) + + def test_appearance_ref_pattern(self): + ok = self.base() + ok["appearances"] = {"mishnah": ["shekalim 4:2"]} + self.assertTrue(is_valid("person", ok)) + bad = self.base() + bad["appearances"] = {"mishnah": ["Shekalim chapter 4"]} + self.assertFalse(is_valid("person", bad)) + + +class TestPlace(unittest.TestCase): + def base(self): + return {"slug": "x", "status": "stub", "type": "settlement", "names": {"he": "פ"}} + + def test_minimal_valid(self): + self.assertTrue(is_valid("place", self.base())) + + def test_geo_null_allowed(self): + d = self.base() + d["geo"] = None # temple_structure / legal_domain + self.assertTrue(is_valid("place", d)) + + def test_coordinate_claim_requires_source(self): + d = self.base() + d["geo"] = {"coordinates": [{"value": {"lat": 1, "lon": 2}}]} + self.assertFalse(is_valid("place", d)) + + def test_geo_consensus_enum(self): + d = self.base() + d["geo"] = {"consensus": "definitely"} + self.assertFalse(is_valid("place", d)) + + +class TestPlant(unittest.TestCase): + def base(self): + return {"slug": "x", "status": "stub", "term": {"he": "פ"}, "term_type": "species"} + + def test_minimal_valid(self): + self.assertTrue(is_valid("plant", self.base())) + + def test_term_type_enum(self): + d = self.base() + d["term_type"] = "weed" + self.assertFalse(is_valid("plant", d)) + + def test_candidate_requires_source(self): + d = self.base() + d["identification"] = {"candidates": [{"value": {"taxon": "T", "rank": "species"}}]} + self.assertFalse(is_valid("plant", d)) + + def test_candidate_valid_with_source_and_ids(self): + d = self.base() + d["identification"] = {"candidates": [ + {"value": {"taxon": "T", "rank": "species"}, "source": "feliks", + "ids": {"wikidata_qid": "Q1"}}]} + self.assertTrue(is_valid("plant", d)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_d3_registries.py b/tests/test_d3_registries.py new file mode 100644 index 0000000..14414b5 --- /dev/null +++ b/tests/test_d3_registries.py @@ -0,0 +1,56 @@ +"""Deliverable 3: source registry + controlled vocabularies.""" + +import unittest + +from _util import DATA, load_yaml, is_valid, masechot_slugs, sedarim_slugs + + +class TestSources(unittest.TestCase): + def test_sources_validate(self): + self.assertTrue(is_valid("source", load_yaml(DATA / "sources.yaml"))) + + def test_sefaria_is_registered(self): + self.assertIn("sefaria", load_yaml(DATA / "sources.yaml")) + + +class TestSedarim(unittest.TestCase): + def test_count_is_63(self): + self.assertEqual(len(sedarim_slugs()), 63) + + def test_slugs_match_masechot_html_files(self): + self.assertEqual(sedarim_slugs(), masechot_slugs()) + + def test_chapters_are_positive_ints(self): + data = load_yaml(DATA / "vocab" / "sedarim.yaml") + for masechtot in data.values(): + for m in masechtot: + self.assertIsInstance(m["chapters"], int) + self.assertGreater(m["chapters"], 0) + + +class TestVocabParse(unittest.TestCase): + def test_all_vocab_files_parse(self): + for name in ("generations", "halachic-categories", "regions", "sedarim"): + with self.subTest(vocab=name): + self.assertIsInstance(load_yaml(DATA / "vocab" / f"{name}.yaml"), dict) + + +class TestExemplarVocabRefs(unittest.TestCase): + """The hand-written exemplars must reference real vocabulary entries.""" + + def test_chitah_halachic_in_vocab(self): + cats = set(load_yaml(DATA / "vocab" / "halachic-categories.yaml")) + used = set(load_yaml(DATA / "plants" / "chitah.yaml")["usage"]["halachic"]) + self.assertTrue(used <= cats, f"unknown halachic categories: {used - cats}") + + def test_tzippori_region_in_vocab(self): + regions = set(load_yaml(DATA / "vocab" / "regions.yaml")) + self.assertIn(load_yaml(DATA / "places" / "tzippori.yaml")["geo"]["region"], regions) + + def test_akiva_generation_in_vocab(self): + gens = set(load_yaml(DATA / "vocab" / "generations.yaml")) + self.assertIn(load_yaml(DATA / "people" / "akiva.yaml")["era"]["generation"], gens) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_d4_validator.py b/tests/test_d4_validator.py new file mode 100644 index 0000000..6845a99 --- /dev/null +++ b/tests/test_d4_validator.py @@ -0,0 +1,63 @@ +"""Deliverable 4: kb-validate.py (schema + semantic cross-checks, exit codes).""" + +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from _util import DATA, SCHEMA, SCRIPTS + + +def run_validate(data_dir): + return subprocess.run( + [sys.executable, str(SCRIPTS / "kb-validate.py"), + "--data", str(data_dir), "--schema", str(SCHEMA)], + capture_output=True, text=True) + + +class TestValidatorOnRealData(unittest.TestCase): + def test_real_data_passes(self): + r = run_validate(DATA) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + + +class TestValidatorCatchesFailures(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + self.data = self.tmp / "data" + shutil.copytree(DATA, self.data) + + def tearDown(self): + shutil.rmtree(self.tmp) + + def _edit(self, rel, old, new): + p = self.data / rel + p.write_text(p.read_text().replace(old, new)) + + def assert_fails_with(self, needle): + r = run_validate(self.data) + self.assertEqual(r.returncode, 1, "expected failure exit") + self.assertIn(needle, r.stdout) + + def test_schema_violation(self): + self._edit("people/akiva.yaml", "status: enriched", "status: bogus") + self.assert_fails_with("is not one of") + + def test_unknown_source(self): + self._edit("plants/chitah.yaml", "source: feliks", "source: nonesuch") + self.assert_fails_with("is not in sources.yaml") + + def test_noncanonical_appearance_slug(self): + self._edit("places/tzippori.yaml", " mishnah: []", ' mishnah: ["notamasechet 1:1"]') + self.assert_fails_with("is not a canonical masechet slug") + + def test_filename_slug_mismatch(self): + shutil.move(self.data / "people" / "akiva.yaml", + self.data / "people" / "rabbi-akiva.yaml") + self.assert_fails_with("does not match filename") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_d5_exemplars.py b/tests/test_d5_exemplars.py new file mode 100644 index 0000000..cb43f6f --- /dev/null +++ b/tests/test_d5_exemplars.py @@ -0,0 +1,39 @@ +"""Deliverable 5: the hand-written exemplars (akiva, tzippori, chitah).""" + +import unittest + +from _util import DATA, load_yaml, validator_for, sedarim_slugs + +EXEMPLARS = {"person": "people/akiva.yaml", + "place": "places/tzippori.yaml", + "plant": "plants/chitah.yaml"} + + +class TestExemplars(unittest.TestCase): + def test_each_validates_against_its_schema(self): + for kind, rel in EXEMPLARS.items(): + with self.subTest(kind=kind): + errs = list(validator_for(kind).iter_errors(load_yaml(DATA / rel))) + self.assertEqual(errs, [], [e.message for e in errs]) + + def test_filename_matches_slug(self): + for rel in EXEMPLARS.values(): + with self.subTest(file=rel): + doc = load_yaml(DATA / rel) + self.assertEqual(doc["slug"], (DATA / rel).stem) + + def test_appearances_use_canonical_slugs(self): + slugs = sedarim_slugs() + for rel in EXEMPLARS.values(): + doc = load_yaml(DATA / rel) + for ref in (doc.get("appearances") or {}).get("mishnah") or []: + with self.subTest(ref=ref): + self.assertIn(ref.rsplit(" ", 1)[0], slugs) + + def test_one_exemplar_per_kind_exists(self): + for rel in EXEMPLARS.values(): + self.assertTrue((DATA / rel).exists(), rel) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_d6_build.py b/tests/test_d6_build.py new file mode 100644 index 0000000..014e942 --- /dev/null +++ b/tests/test_d6_build.py @@ -0,0 +1,86 @@ +"""Deliverable 6: kb-build.py (YAML -> knowledge.db).""" + +import json +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from _util import DATA, load_script, sedarim_slugs + +kb_build = load_script("kb_build", "kb-build.py") + + +class BuildCase(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.tmp = Path(tempfile.mkdtemp()) + cls.db_path = cls.tmp / "knowledge.db" + cls.counts = kb_build.build(DATA, cls.db_path) + cls.db = sqlite3.connect(cls.db_path) + + @classmethod + def tearDownClass(cls): + cls.db.close() + + def q(self, sql, *a): + return self.db.execute(sql, a).fetchall() + + +class TestStructure(BuildCase): + def test_entity_row_per_yaml_file(self): + n_files = sum(len(list((DATA / f).glob("*.yaml"))) + for f in ("people", "places", "plants")) + self.assertEqual(self.counts["entity"], n_files) + + def test_source_table_matches_registry(self): + import yaml + with open(DATA / "sources.yaml") as f: + self.assertEqual(self.counts["source"], len(yaml.safe_load(f))) + + def test_every_claim_source_is_registered(self): + bad = self.q("SELECT field FROM claim WHERE source NOT IN (SELECT key FROM source)") + self.assertEqual(bad, []) + + def test_mishnah_appearances_parse_to_canonical_masechet(self): + slugs = sedarim_slugs() + for ref, masechet in self.q("SELECT ref, masechet FROM appearance WHERE work='mishnah'"): + self.assertEqual(masechet, ref.rsplit(" ", 1)[0]) + self.assertIn(masechet, slugs) + + def test_other_appearances_have_no_masechet(self): + rows = self.q("SELECT masechet FROM appearance WHERE work='other'") + self.assertTrue(all(m is None for (m,) in rows)) + + def test_external_ids_have_no_nulls(self): + self.assertEqual(self.q("SELECT * FROM external_id WHERE id IS NULL"), []) + + +class TestFlattening(BuildCase): + def test_candidate_ids_preserved_in_extra_json(self): + (extra,) = self.q( + "SELECT extra_json FROM claim WHERE entity_slug='chitah' AND source='feliks'")[0] + self.assertIn("wikidata_qid", json.loads(extra)["ids"]) + + def test_coordinate_value_is_lat_lon(self): + (vj,) = self.q( + "SELECT value_json FROM claim WHERE entity_slug='tzippori' " + "AND field='geo.coordinates'")[0] + self.assertEqual(set(json.loads(vj)), {"lat", "lon"}) + + def test_region_rollup(self): + rows = dict(self.q("SELECT region, place_count FROM region_rollup")) + self.assertEqual(rows.get("galilee"), 1) + + +class TestDeterminism(unittest.TestCase): + def test_build_is_byte_identical_across_runs(self): + tmp = Path(tempfile.mkdtemp()) + a, b = tmp / "a.db", tmp / "b.db" + kb_build.build(DATA, a) + kb_build.build(DATA, b) + self.assertEqual(a.read_bytes(), b.read_bytes()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_d7_importers.py b/tests/test_d7_importers.py new file mode 100644 index 0000000..625d4bf --- /dev/null +++ b/tests/test_d7_importers.py @@ -0,0 +1,185 @@ +"""Deliverable 7: the §8 upsert engine (kb_lib) and the importers.""" + +import json +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from _util import DATA, FIXTURES, SCRIPTS, load_script # puts scripts/ on sys.path + +import kb_lib as kb +from ruamel.yaml.comments import CommentedMap + +wd_import = load_script("kb_import_wikidata", "kb-import-wikidata.py") +pl_import = load_script("kb_import_pleiades", "kb-import-pleiades.py") + + +def entity(**kw): + d = CommentedMap() + d.update(kw) + return d + + +def load_fixture(rel): + return json.loads((FIXTURES / rel).read_text()) + + +# --------------------------------------------------------------------------- # +# §8 merge rules +# --------------------------------------------------------------------------- # +class TestEngine(unittest.TestCase): + def test_single_reupsert_same_value_unchanged(self): + d, c = entity(slug="x"), [] + kb.upsert_claim(d, "bio", "A", "wikidata", mode="single", conflicts=c) + r = kb.upsert_claim(d, "bio", "A", "wikidata", mode="single", conflicts=c) + self.assertEqual(r, "unchanged") + self.assertEqual(len(d["bio"]), 1) + + def test_single_changed_value_updates_in_place(self): + d, c = entity(slug="x"), [] + kb.upsert_claim(d, "bio", "A", "wikidata", mode="single", conflicts=c) + r = kb.upsert_claim(d, "bio", "B", "wikidata", mode="single", conflicts=c) + self.assertEqual(r, "updated") + self.assertEqual(len(d["bio"]), 1) + self.assertEqual(d["bio"][0]["value"], "B") + + def test_confirmed_claim_is_protected(self): + d, c = entity(slug="x"), [] + kb.upsert_claim(d, "bio", "keep", "manual", mode="single", conflicts=c, confirmed=True) + r = kb.upsert_claim(d, "bio", "overwrite", "manual", mode="single", conflicts=c) + self.assertEqual(r, "conflict") + self.assertEqual(d["bio"][0]["value"], "keep") + self.assertEqual(len(c), 1) + + def test_multi_dedups_and_retains_dissent(self): + d, c = entity(slug="x"), [] + kb.upsert_claim(d, "geo.coordinates", {"lat": 1, "lon": 2}, "pleiades", mode="multi", conflicts=c) + same = kb.upsert_claim(d, "geo.coordinates", {"lat": 1, "lon": 2}, "pleiades", mode="multi", conflicts=c) + other = kb.upsert_claim(d, "geo.coordinates", {"lat": 9, "lon": 9}, "wikidata", mode="multi", conflicts=c) + self.assertEqual(same, "unchanged") + self.assertEqual(other, "added") + self.assertEqual(len(d["geo"]["coordinates"]), 2) + + def test_appearance_dedupe(self): + d = entity(slug="x") + self.assertTrue(kb.add_appearance(d, "mishnah", "shekalim 4:2")) + self.assertFalse(kb.add_appearance(d, "mishnah", "shekalim 4:2")) + + def test_external_id_idempotent(self): + d = entity(slug="x") + self.assertTrue(kb.set_external_id(d, "wikidata_qid", "Q1")) + self.assertFalse(kb.set_external_id(d, "wikidata_qid", "Q1")) + + def test_flow_map_renders_inline(self): + d = entity(slug="x") + kb.upsert_claim(d, "geo.coordinates", kb.flow_map({"lat": 1, "lon": 2}), + "pleiades", mode="multi", conflicts=[]) + import io + buf = io.StringIO() + kb.make_yaml().dump(d, buf) + self.assertIn("{lat: 1, lon: 2}", buf.getvalue()) + + def test_roundtrip_preserves_comments(self): + data = kb.load_entity(DATA / "people" / "akiva.yaml") + kb.upsert_claim(data, "bio", "new note", "manual", mode="multi", conflicts=[]) + import io + buf = io.StringIO() + kb.make_yaml().dump(data, buf) + self.assertIn("tannaitic generation", buf.getvalue()) + + +# --------------------------------------------------------------------------- # +# Wikidata importer transforms +# --------------------------------------------------------------------------- # +class TestWikidataTransform(unittest.TestCase): + def test_person_bio_and_variants(self): + d, c = entity(slug="akiva", names=CommentedMap({"he": "רבי עקיבא"})), [] + wd = load_fixture("wikidata/Q310357.json") + wd_import.transform("person", "Q310357", wd, d, c) + self.assertEqual(d["bio"][0]["source"], "wikidata") + self.assertIn("tanna", d["bio"][0]["value"]) + self.assertIn("Akiva ben Joseph", d["names"]["variants"]) + + def test_person_is_idempotent(self): + d, c = entity(slug="akiva", names=CommentedMap({"he": "רבי עקיבא"})), [] + wd = load_fixture("wikidata/Q310357.json") + wd_import.transform("person", "Q310357", wd, d, c) + log = wd_import.transform("person", "Q310357", wd, d, c) + self.assertTrue(all(s in ("unchanged", "0") for _, s in log), log) + self.assertEqual(len(d["bio"]), 1) + + def test_place_coord_and_photo(self): + d, c = entity(slug="tzippori", geo=CommentedMap({"region": "galilee"})), [] + wd = load_fixture("wikidata/Q745966.json") + wd_import.transform("place", "Q745966", wd, d, c) + coord = d["geo"]["coordinates"][0]["value"] + self.assertEqual((coord["lat"], coord["lon"]), (32.7522, 35.2797)) + self.assertIn("commons.wikimedia.org", d["media"]["photo"]) + + def test_plant_candidate_via_qid(self): + d, c = entity(slug="chitah", term=CommentedMap({"he": "חיטה"})), [] + wd = load_fixture("wikidata/Q12100.json") + wd_import.transform("plant", "Q12100", wd, d, c) + cand = d["identification"]["candidates"][0] + self.assertEqual(cand["value"]["taxon"], "Triticum aestivum") + self.assertEqual(cand["value"]["rank"], "species") # P105 Q7432 -> species + self.assertEqual(cand["ids"]["wikidata_qid"], "Q12100") + + def test_confirmed_bio_protected_through_importer(self): + d, c = entity(slug="akiva", names=CommentedMap({"he": "x"})), [] + kb.upsert_claim(d, "bio", "hand-written", "wikidata", mode="single", + conflicts=c, confirmed=True) + wd = load_fixture("wikidata/Q310357.json") + wd_import.transform("person", "Q310357", wd, d, c) + self.assertEqual(d["bio"][0]["value"], "hand-written") + self.assertEqual(len(c), 1) + + +# --------------------------------------------------------------------------- # +# Pleiades importer transform +# --------------------------------------------------------------------------- # +class TestPleiadesTransform(unittest.TestCase): + def test_coord_from_repr_point(self): + d, c = entity(slug="tzippori", geo=CommentedMap({})), [] + pl = load_fixture("pleiades/678378.json") + pl_import.transform(pl, d, c) + coord = d["geo"]["coordinates"][0]["value"] + self.assertEqual((coord["lat"], coord["lon"]), (32.7522, 35.2797)) # reprPoint is [lon, lat] + + +# --------------------------------------------------------------------------- # +# CLI integration: writes valid YAML, flow style, idempotent +# --------------------------------------------------------------------------- # +class TestImporterCLI(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + self.data = self.tmp / "data" + shutil.copytree(DATA, self.data) + + def tearDown(self): + shutil.rmtree(self.tmp) + + def run_wd(self, *args): + return subprocess.run( + [sys.executable, str(SCRIPTS / "kb-import-wikidata.py"), + "--data", str(self.data), *args], + capture_output=True, text=True) + + def test_place_import_writes_flow_coord_and_is_idempotent(self): + first = self.run_wd("--kind", "places", "--slug", "tzippori", + "--input", str(FIXTURES / "wikidata/Q745966.json")) + self.assertEqual(first.returncode, 0, first.stderr) + self.assertIn("geo.coordinates=added", first.stdout) + text = (self.data / "places" / "tzippori.yaml").read_text() + self.assertIn("{lat: 32.7522, lon: 35.2797}", text) # flow style preserved + # re-run -> no change + second = self.run_wd("--kind", "places", "--slug", "tzippori", + "--input", str(FIXTURES / "wikidata/Q745966.json")) + self.assertIn("no change", second.stdout) + + +if __name__ == "__main__": + unittest.main() From 29b300d24abe52786f3a9643b0a20f639305f2d9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 29 Jun 2026 11:01:50 +0000 Subject: [PATCH 7/7] =?UTF-8?q?Display=20phase=202:=20kb-render.py=20?= =?UTF-8?q?=E2=80=94=20static=20entity/index/map/gallery=20views?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scripts/kb-render.py: renders entities/site/ from the YAML source of truth (index + who's-who + places map + flora gallery + one page per entity) - house-style self-contained HTML; contested candidates/coordinates stay visible; appearances deep-link into masechot/{slug}.html#mishna-N-M - reads YAML (full fidelity) not the lossy query DB; reconciled display spec §4 - tests/test_render.py (7); entities/site/ gitignored - 77 tests total green --- .gitignore | 1 + README.md | 1 + docs/entities-display.md | 2 +- scripts/kb-render.py | 385 +++++++++++++++++++++++++++++++++++++++ tests/test_render.py | 66 +++++++ 5 files changed, 454 insertions(+), 1 deletion(-) create mode 100644 scripts/kb-render.py create mode 100644 tests/test_render.py diff --git a/.gitignore b/.gitignore index 7bb0a41..03ee5d1 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ output/ # Entities KB derived artifacts (rebuilt from entities/data/ — never committed) entities/knowledge.db entities/conflicts.log +entities/site/ diff --git a/README.md b/README.md index b078583..e11d59b 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ This repo contains: - **Entities knowledge base** (`entities/`) — an accumulating store of people, places, and plants in the Mishnah ([KB spec](docs/entities-knowledge-base.md), [display spec](docs/entities-display.md)). YAML is the source of truth (run `pip install -r entities/requirements.txt` first): - `scripts/kb-validate.py` — validate `entities/data/` against the JSON schemas + semantic cross-checks - `scripts/kb-build.py` — compile the YAML into the derived `entities/knowledge.db` + - `scripts/kb-render.py` — render the static entity/index/map/gallery site into `entities/site/` (display phase 2) - `scripts/kb-import-wikidata.py`, `scripts/kb-import-pleiades.py` — enrich entities from external sources (additive, idempotent; `--input` runs offline against `entities/fixtures/`) - `scripts/kb-selftest.py` — quick offline smoke of the §8 merge-rule invariants - `tests/` — full suite, one module per deliverable: `python3 -m unittest discover -s tests -t tests` diff --git a/docs/entities-display.md b/docs/entities-display.md index 25c1ed6..fc2bb72 100644 --- a/docs/entities-display.md +++ b/docs/entities-display.md @@ -60,7 +60,7 @@ masechot/*.html + DB ──kb-enrich─▶ enriched masechot (3) mark mishn The order matters and is intentional: - **Phase 1 — `scripts/kb-build.py`** compiles the YAML into `knowledge.db` (per the KB spec §11). Nothing here is presentational. -- **Phase 2 — `scripts/kb-render.py`** generates the standalone, DB-only views: one **entity page** per person/place/plant (§5), plus aggregate views — a **places map**, a **flora gallery**, and a **who's-who / index**. These never contain mishna text; they are pure projections of the KB. +- **Phase 2 — `scripts/kb-render.py`** generates the standalone views: one **entity page** per person/place/plant (§5), plus aggregate views — a **places map**, a **flora gallery**, and a **who's-who / index**. These never contain mishna text; they are pure projections of the KB. (Implementation note: it renders from the **YAML source of truth** for full fidelity — `knowledge.db`'s §11 schema is intentionally a minimal query index and drops scalar fields like `region`, `generation`, and `halachic`. The DB stays the cross-entity query layer; per-page rendering reads the richer YAML.) - **Phase 3 — `scripts/kb-enrich.py`** is the *join*, and therefore last: it takes a canonical masechet plus the DB and produces an **enriched copy** in which entity mentions in the running text are wrapped, colored (under the overlay), and linked to the phase-2 entity pages. It depends on phase 2 (the link targets must exist) and on the appearance linkage (to know which mishnayot mention what). All phase-2 and phase-3 outputs are derived artifacts: gitignored, regenerable, output to a build/publish directory rather than committed over `masechot/`. diff --git a/scripts/kb-render.py b/scripts/kb-render.py new file mode 100644 index 0000000..96ee147 --- /dev/null +++ b/scripts/kb-render.py @@ -0,0 +1,385 @@ +#!/usr/bin/env python3 +"""Render the entities knowledge base into a static HTML site (display phase 2). + +Usage: + python3 scripts/kb-render.py + python3 scripts/kb-render.py --data entities/data --out entities/site + +Pure views on the KB (docs/entities-display.md §4–5): one page per entity, plus +aggregate views — a who's-who index (people), a map (places), and a flora gallery +(plants). These never contain mishna text; appearances are deep links into the +canonical masechot via the anchor scheme in docs/html-format.md. + +Reads the YAML source of truth (full fidelity); knowledge.db remains the +cross-entity query index per KB spec §11. Output is derived & gitignored. + +Requires: pyyaml. +""" + +import argparse +import html +import sys +from pathlib import Path + +try: + import yaml +except ImportError: + sys.exit("kb-render: PyYAML is required — pip install pyyaml") + +ROOT = Path(__file__).resolve().parent.parent +KINDS = {"people": "person", "places": "place", "plants": "plant"} +KIND_LABEL = {"people": "People", "places": "Places", "plants": "Plants"} +KIND_ICON = {"people": "👤", "places": "📍", "plants": "🌿"} + +CSS = """ +:root { --ink:#1a1a1a; --bg:#fdfdfa; --link:#2a5a8a; --muted:#6b6b66; --rule:#e6e6df; } +* { box-sizing: border-box; } +body { background: var(--bg); color: var(--ink); margin: 0; + font-family: Georgia, "Times New Roman", serif; line-height: 1.6; } +.he { font-family: "SBL Hebrew","Frank Ruehl CLM","Ezra SIL","David CLM", + "Noto Serif Hebrew","Times New Roman",serif; direction: rtl; } +main { max-width: 46em; margin: 0 auto; padding: 1.5rem 1.25rem 4rem; } +a { color: var(--link); text-decoration: none; } +a:hover { text-decoration: underline; } +nav.crumbs { font-size: .85rem; color: var(--muted); margin-bottom: 1.5rem; } +h1 { font-size: 2rem; margin: 0 0 .15rem; } +h1 .en { display:block; font-size: 1rem; color: var(--muted); font-weight: normal; } +h2 { font-size: 1.1rem; border-bottom: 1px solid var(--rule); padding-bottom: .25rem; + margin: 2rem 0 .75rem; } +.tag { display:inline-block; font-size:.72rem; padding:.05rem .45rem; border-radius:1rem; + background:#eef1f4; color:#3a4a5a; margin-left:.25rem; } +.conf { font-size:.7rem; color:var(--muted); } +.conf.accepted{color:#2e7d32;} .conf.minority,.conf.disputed{color:#b26a00;} +.src { font-size:.7rem; color:var(--muted); } +ul.bare { list-style:none; padding:0; margin:.3rem 0; } +ul.bare li { margin:.25rem 0; } +.meta { color:var(--muted); font-size:.9rem; margin:.25rem 0 1rem; } +.claim { margin:.4rem 0; padding:.4rem .6rem; border-left:3px solid var(--rule); } +.claim.primary { border-left-color: var(--link); } +table { border-collapse: collapse; width:100%; font-size:.95rem; } +th,td { text-align:start; padding:.4rem .5rem; border-bottom:1px solid var(--rule); vertical-align:top; } +th { font-size:.8rem; color:var(--muted); text-transform:uppercase; letter-spacing:.03em; } +.gallery { display:grid; grid-template-columns:repeat(auto-fill,minmax(11rem,1fr)); gap:1rem; } +.card { border:1px solid var(--rule); border-radius:.5rem; padding:.9rem; } +.card .taxon { font-style:italic; color:var(--muted); font-size:.85rem; } +.home { display:grid; grid-template-columns:repeat(3,1fr); gap:1rem; margin-top:2rem; } +.home a { border:1px solid var(--rule); border-radius:.5rem; padding:1.25rem; text-align:center; + color:var(--ink); font-size:1.1rem; } +.home a .n { display:block; font-size:2rem; color:var(--link); } +footer { color:var(--muted); font-size:.8rem; border-top:1px solid var(--rule); + margin-top:3rem; padding-top:1rem; } +""" + + +def esc(x): + return html.escape(str(x)) if x is not None else "" + + +def he(text): + return f'{esc(text)}' + + +def page(title, body, out_root): + """out_root: relative prefix from this page's dir to entities/site/.""" + return f""" + + + + +{esc(title)} + + + +
+ +{body} +
Generated by kb-render.py — a view on the entities knowledge base. +Source of truth: entities/data/.
+
+ + +""" + + +def appearances_html(doc, masechot_prefix): + app = doc.get("appearances") or {} + mish = app.get("mishnah") or [] + other = app.get("other") or [] + if not mish and not other: + return "

No recorded appearances yet.

" + out = [] + if mish: + links = [] + for ref in mish: + slug, _, cm = ref.partition(" ") + ch, _, mi = cm.partition(":") + href = f"{masechot_prefix}masechot/{slug}.html#mishna-{ch}-{mi}" + links.append(f'{esc(ref)}') + out.append("
    " + "".join(f"
  • {l}
    • " for l in links) + "
") + if other: + out.append("

Other: " + ", ".join(esc(o) for o in other) + "

") + return "\n".join(out) + + +def claim_html(value_html, claim, primary=False): + conf = claim.get("confidence") + src = claim.get("source") + by = claim.get("asserted_by") + bits = [] + if conf: + bits.append(f'{esc(conf)}') + if by: + bits.append(f'{esc(by)}') + if src: + bits.append(f'[{esc(src)}]') + cls = "claim primary" if primary else "claim" + return f'
{value_html}   {" ".join(bits)}
' + + +def entity_link(reg, kind, slug): + """Link to another entity page if it exists, else show the bare slug.""" + doc = reg.get(kind, {}).get(slug) + if doc: + name = (doc.get("names") or {}).get("he") or (doc.get("term") or {}).get("he") or slug + folder = {"person": "people", "place": "places", "plant": "plants"}[kind] + return f'{he(name)} {esc(slug)}' + return f'{esc(slug)}' + + +# --------------------------------------------------------------------------- # +# per-entity pages (live at site//.html -> masechot at ../../../) +# --------------------------------------------------------------------------- # +MASECHOT_FROM_ENTITY = "../../../" +SITE_FROM_ENTITY = "../../" + + +def render_person(doc, reg): + names = doc.get("names") or {} + era = doc.get("era") or {} + rel = doc.get("relationships") or {} + parts = [f'

{he(names.get("he"))}{esc(names.get("en"))}' + f'{esc(doc.get("type"))}' + f'{esc(doc.get("status"))}

'] + meta = [] + if era.get("generation"): + meta.append(f"Generation {esc(era['generation'])}") + if era.get("floruit"): + meta.append(esc(era["floruit"])) + if meta: + parts.append(f'

{" · ".join(meta)}

') + if names.get("variants"): + parts.append("

Also: " + + ", ".join(he(v) for v in names["variants"]) + "

") + for label, key in (("Teachers", "teachers"), ("Students", "students"), ("Family", "family")): + if rel.get(key): + parts.append(f"

{label}

    " + + "".join(f"
  • {entity_link(reg, 'person', s)}
    • " for s in rel[key]) + + "
") + if doc.get("bio"): + parts.append("

Biography

") + for c in doc["bio"]: + parts.append(claim_html(esc(c.get("value")), c)) + parts.append("

Appearances

" + appearances_html(doc, MASECHOT_FROM_ENTITY)) + parts.append(ids_html(doc)) + return "\n".join(parts) + + +def render_place(doc, reg): + names = doc.get("names") or {} + geo = doc.get("geo") or {} + parts = [f'

{he(names.get("he"))}{esc(names.get("en"))}' + f'{esc(doc.get("type"))}' + f'{esc(doc.get("status"))}

'] + extra = [] + if names.get("classical"): + extra.append("Classical: " + ", ".join(esc(c) for c in names["classical"])) + if geo.get("region"): + extra.append("Region: " + esc(geo["region"])) + if geo.get("modern_admin"): + extra.append(esc(geo["modern_admin"])) + if geo.get("consensus"): + extra.append("Identification: " + esc(geo["consensus"])) + if extra: + parts.append(f'

{" · ".join(extra)}

') + coords = geo.get("coordinates") or [] + if coords: + parts.append("

Coordinates

") + for i, c in enumerate(coords): + v = c.get("value") or {} + osm = f'https://www.openstreetmap.org/?mlat={v.get("lat")}&mlon={v.get("lon")}' + val = f'{esc(v.get("lat"))}, {esc(v.get("lon"))}  map ↗' + parts.append(claim_html(val, c, primary=(i == 0))) + media = doc.get("media") or {} + if media.get("photo"): + parts.append(f'

Photo ↗

') + parts.append("

Appearances

" + appearances_html(doc, MASECHOT_FROM_ENTITY)) + parts.append(ids_html(doc)) + return "\n".join(parts) + + +def render_plant(doc, reg): + term = doc.get("term") or {} + ident = doc.get("identification") or {} + names = doc.get("names") or {} + usage = doc.get("usage") or {} + parts = [f'

{he(term.get("he"))}{esc(term.get("en_common"))}' + f'{esc(doc.get("term_type"))}' + f'{esc(doc.get("status"))}

'] + if term.get("variants"): + parts.append("

Forms: " + + ", ".join(he(v) for v in term["variants"]) + "

") + parts.append("

Identification

") + if ident.get("display_taxon"): + parts.append(f'

Displayed as {esc(ident["display_taxon"])} ' + f'({esc(ident.get("consensus","?"))})

') + for i, c in enumerate(ident.get("candidates") or []): + v = c.get("value") or {} + val = f'{esc(v.get("taxon"))} {esc(v.get("rank"))}' + parts.append(claim_html(val, c, primary=(i == 0))) + nrows = [] + for label, key in (("Botanical", "botanical"), ("Family", "family")): + if names.get(key): + nrows.append(f"{label}: {esc(names[key])}") + for label, key in (("English", "common_en"), ("Modern Hebrew", "modern_he"), ("Arabic", "arabic")): + if names.get(key): + joined = ", ".join(he(x) if key != "common_en" else esc(x) for x in names[key]) + nrows.append(f"{label}: {joined}") + if nrows: + parts.append("

Names

    " + + "".join(f"
  • {r}
    • " for r in nrows) + "
") + if usage.get("halachic"): + parts.append("

Halachic categories

" + + " ".join(f'{esc(t)}' for t in usage["halachic"]) + "

") + media = doc.get("media") or {} + mlinks = [f'{lbl} ↗' + for k, lbl in (("photo", "Photo"), ("illustration", "Illustration")) if media.get(k)] + if mlinks: + parts.append('

' + " · ".join(mlinks) + "

") + parts.append("

Appearances

" + appearances_html(doc, MASECHOT_FROM_ENTITY)) + parts.append(ids_html(doc)) + return "\n".join(parts) + + +ID_LABELS = {"wikidata_qid": ("Wikidata", "https://www.wikidata.org/wiki/{}"), + "pleiades_id": ("Pleiades", "https://pleiades.stoa.org/places/{}"), + "sefaria_slug": ("Sefaria", None)} + + +def ids_html(doc): + ids = {k: v for k, v in (doc.get("ids") or {}).items() if v} + if not ids: + return "" + links = [] + for k, v in ids.items(): + label, tpl = ID_LABELS.get(k, (k, None)) + if tpl: + links.append(f'{esc(label)} ↗') + else: + links.append(f"{esc(label)}: {esc(v)}") + return "

External IDs

" + " · ".join(links) + "

" + + +RENDERERS = {"person": render_person, "place": render_place, "plant": render_plant} + + +# --------------------------------------------------------------------------- # +# aggregate pages (live at site/.html -> masechot at ../../, entities at /) +# --------------------------------------------------------------------------- # +def render_people_index(reg): + rows = [] + for slug, doc in sorted(reg["person"].items()): + names = doc.get("names") or {} + gen = (doc.get("era") or {}).get("generation") or "" + n = len((doc.get("appearances") or {}).get("mishnah") or []) + rows.append(f"{he(names.get('he'))}" + f"{esc(names.get('en'))}{esc(doc.get('type'))}" + f"{esc(gen)}{n}") + return ("

Who's Who 👤

" + "" + "".join(rows) + "
NameEnglishTypeGen.Mishnayot
") + + +def render_places_index(reg): + rows = [] + for slug, doc in sorted(reg["place"].items()): + names = doc.get("names") or {} + geo = doc.get("geo") or {} + coords = (geo.get("coordinates") or [{}])[0].get("value") or {} + osm = (f"{esc(coords.get('lat'))}, {esc(coords.get('lon'))} ↗" + if coords else "") + rows.append(f"{he(names.get('he'))}" + f"{esc(names.get('en'))}{esc(geo.get('region'))}" + f"{osm}") + return ("

Places 📍

A static index; an interactive map is a later " + "enhancement (display spec §9).

" + "" + "".join(rows) + "
NameEnglishRegionCoordinates
") + + +def render_plants_index(reg): + cards = [] + for slug, doc in sorted(reg["plant"].items()): + term = doc.get("term") or {} + taxon = (doc.get("identification") or {}).get("display_taxon") or "" + cards.append(f"" + f"
{he(term.get('he'))}
" + f"
{esc(term.get('en_common'))}
" + f"
{esc(taxon)}
") + return "

Flora Gallery 🌿

" + "".join(cards) + "
" + + +AGG = {"people": render_people_index, "places": render_places_index, "plants": render_plants_index} + + +def render_home(reg): + cells = [] + for folder, kind in KINDS.items(): + cells.append(f"{len(reg[kind])}" + f"{KIND_ICON[folder]} {KIND_LABEL[folder]}") + return ("

Mishnah Entities

" + "

People, places, and plants of the Mishnah — a view on the " + "entities knowledge base.

" + "".join(cells) + "
") + + +def load_all(data_dir): + reg = {"person": {}, "place": {}, "plant": {}} + for folder, kind in KINDS.items(): + sub = Path(data_dir) / folder + for path in sorted(sub.glob("*.yaml")): + with open(path, encoding="utf-8") as f: + reg[kind][path.stem] = yaml.safe_load(f) + return reg + + +def main(): + ap = argparse.ArgumentParser(description="Render the entities KB to a static site.") + ap.add_argument("--data", default=str(ROOT / "entities" / "data")) + ap.add_argument("--out", default=str(ROOT / "entities" / "site")) + args = ap.parse_args() + + reg = load_all(args.data) + out = Path(args.out) + written = 0 + + def write(rel, body, out_root): + nonlocal written + title_map = {"index.html": "Mishnah Entities"} + title = title_map.get(rel, rel) + p = out / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(page(title, body, out_root), encoding="utf-8") + written += 1 + + write("index.html", render_home(reg), "") + for folder, kind in KINDS.items(): + write(f"{folder}.html", AGG[folder](reg), "") + for slug, doc in sorted(reg[kind].items()): + write(f"{folder}/{slug}.html", RENDERERS[kind](doc, reg), SITE_FROM_ENTITY) + + print(f"rendered {written} page(s) -> {out}") + for folder, kind in KINDS.items(): + print(f" {folder:<8} {len(reg[kind])} entit(ies)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_render.py b/tests/test_render.py new file mode 100644 index 0000000..a33b1bf --- /dev/null +++ b/tests/test_render.py @@ -0,0 +1,66 @@ +"""Display phase 2: kb-render.py (static views on the KB).""" + +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from _util import DATA, SCRIPTS + + +class RenderCase(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.out = Path(tempfile.mkdtemp()) / "site" + r = subprocess.run( + [sys.executable, str(SCRIPTS / "kb-render.py"), + "--data", str(DATA), "--out", str(cls.out)], + capture_output=True, text=True) + assert r.returncode == 0, r.stderr + cls.r = r + + def read(self, rel): + return (self.out / rel).read_text(encoding="utf-8") + + def test_all_pages_written(self): + for rel in ("index.html", "people.html", "places.html", "plants.html", + "people/akiva.html", "places/tzippori.html", "plants/chitah.html"): + with self.subTest(page=rel): + self.assertTrue((self.out / rel).exists(), rel) + + def test_home_links_to_aggregates(self): + home = self.read("index.html") + for agg in ("people.html", "places.html", "plants.html"): + self.assertIn(f"href='{agg}'", home) + + def test_person_page_has_name_and_deep_link(self): + page = self.read("people/akiva.html") + self.assertIn("רבי עקיבא", page) + # appearance resolves to the canonical masechet anchor, correct relative depth + self.assertIn("../../../masechot/brachot.html#mishna-9-5", page) + + def test_plant_page_shows_candidate_with_source(self): + page = self.read("plants/chitah.html") + self.assertIn("Triticum aestivum", page) + self.assertIn("feliks", page) # candidate source visible (dissent stays visible) + self.assertIn("../../../masechot/kilayim.html#mishna-1-1", page) + + def test_place_page_has_coordinate_and_osm(self): + page = self.read("places/tzippori.html") + self.assertIn("openstreetmap.org", page) + self.assertIn("pleiades", page) + + def test_aggregate_links_to_entity_pages(self): + self.assertIn("people/akiva.html", self.read("people.html")) + self.assertIn("plants/chitah.html", self.read("plants.html")) + + def test_nonentity_relationship_falls_back_to_slug(self): + # akiva's teachers aren't entities yet -> rendered as bare slug, not a broken link + page = self.read("people/akiva.html") + self.assertIn("eliezer-b-hyrcanus", page) + self.assertNotIn("people/eliezer-b-hyrcanus.html", page) + + +if __name__ == "__main__": + unittest.main()