feat(skills): reorganize skills/ into domain suites

.gitignore

@@ -40,6 +40,11 @@ local/
 tests/plt_*
 tests/graders/skills/*.json
 
+# Skill functional-test fixtures — tracked despite the *.jsonl rule above;
+# only the fixtures themselves, not generated results/ (ignored below).
+!skills/**/tests/*_test_cases.jsonl
+skills/**/tests/results/
+
 # Security Files
 *.pem
 *.key

cookbooks/skills_evaluation/skill_models.py

@@ -209,7 +209,7 @@ def _read_text(path: Path) -> str:
 class SkillLoader:
     """Loads Agent Skill packages from a directory.
 
-    Supports two directory layouts:
+    Supports individual skills, flat collections, and nested domain suites:
 
     **Single skill**::
 
@@ -225,6 +225,15 @@ class SkillLoader:
             scripts/review.py
           paper-review/
             SKILL.md
+
+    **Domain suites** (grouping directories may be nested)::
+
+        skills_dir/
+          academic-eval/
+            01-paper-review/
+              SKILL.md
+          standalone/
+            SKILL.md
     """
 
     @classmethod
@@ -330,7 +339,10 @@ def load_from_directory(cls, skills_dir: Union[str, Path]) -> List[SkillPackage]
         Args:
             skills_dir: Path to a directory.  If the directory itself contains
                 ``SKILL.md`` it is treated as a single-skill directory; otherwise
-                each immediate subdirectory is checked for a ``SKILL.md``.
+                subdirectories are searched recursively. Discovery stops at
+                any directory containing ``SKILL.md`` so bundled examples are
+                not treated as separate packages. Each resolved directory is
+                visited once, including when reached through a symbolic link.
 
         Returns:
             List of successfully loaded :class:`SkillPackage` objects (may be empty).
@@ -342,20 +354,26 @@ def load_from_directory(cls, skills_dir: Union[str, Path]) -> List[SkillPackage]
         if not skills_dir.is_dir():
             raise ValueError(f"Not a directory: {skills_dir}")
 
-        if (skills_dir / SKILL_MD_NAME).is_file():
-            skill = cls.load_skill(skills_dir)
-            return [skill] if skill else []
-
         skills: List[SkillPackage] = []
-        for subdir in sorted(skills_dir.iterdir()):
-            if not subdir.is_dir():
-                continue
-            if any(p in _IGNORE_DIRS for p in subdir.parts):
-                continue
-            skill = cls.load_skill(subdir)
-            if skill:
-                skills.append(skill)
+        visited: set[Path] = set()
+
+        def collect(directory: Path) -> None:
+            resolved = directory.resolve()
+            if resolved in visited:
+                return
+            visited.add(resolved)
+
+            if (directory / SKILL_MD_NAME).is_file():
+                skill = cls.load_skill(directory)
+                if skill:
+                    skills.append(skill)
+                return
+
+            for subdir in sorted(directory.iterdir()):
+                if subdir.is_dir() and subdir.name not in _IGNORE_DIRS:
+                    collect(subdir)
 
+        collect(skills_dir)
         return skills
 
 

skills/README.md

@@ -0,0 +1,112 @@
+# Skills
+
+This directory holds Agent Skills (Anthropic Agent Skill protocol — YAML
+frontmatter + Markdown body) for OpenJudge. Every `SKILL.md`, wherever it
+lives, is **independently installable**: it carries everything it needs
+inline, so a single `<name>/SKILL.md` folder can be pulled out and installed
+on its own in Claude Code, Cursor, Codex, Hermes, OpenClaw, or any other
+client that speaks the same protocol — you never need this README or a
+suite's own README to make one skill work.
+
+Grouping below **is folder structure only** — there is no code-level registry
+(no `SUITE_REGISTRY`, no `DomainSuite` class). A "domain suite" is just a
+directory of related `<NN-name>/SKILL.md` workflows plus a `README.md` that
+explains how they relate.
+
+## Domain suites (multi-skill, folder-grouped)
+
+| Suite | Skills | Dimension | What it covers |
+|---|---|---|---|
+| [`eval_pipeline/`](eval_pipeline/) | 9 (`00-meta-eval` … `08-bootstrap`) | Horizontal methodology | How to build an evaluation system from scratch — dataset design, metric/grader selection, human-alignment calibration, reporting, plus RAG/prompt-regression/redteam/bootstrap scenarios. Router at `00-meta-eval`. |
+| [`academic-eval/`](academic-eval/) | 4 (`00-academic-router`, `01-paper-review`, `02-bib-verify`, `03-ref-hallucination-arena`) | Vertical domain | Academic paper review, BibTeX verification, and citation-hallucination benchmarking. |
+| [`arena-eval/`](arena-eval/) | 3 (`00-arena-router`, `01-auto-arena`, `02-ref-hallucination-arena`) | Vertical domain | Arena-style model/agent comparison — generic win-rate ranking, and citation-hallucination-specific ranking. |
+| [`openjudge-core/`](openjudge-core/) | 2 (`01-graders-and-pipeline`, `02-rl-reward`) | Horizontal methodology | Direct OpenJudge core-library API: graders/`GradingRunner`/aggregators, and RL reward-signal construction. No router — see the suite's own README for why. |
+
+`eval_pipeline` is the range-of-motion paradigm all the other suites above
+were reshaped to match (three-layer structure, directory-as-grouping, router
+only where sub-skills are genuinely ambiguous, every `<NN-name>/SKILL.md`
+independently installable). See
+[`docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md`](../docs/superpowers/specs/2026-07-09-skills-domain-suite-proposal.md)
+for the full design rationale.
+
+### Functional test coverage
+
+`eval_pipeline`, `academic-eval`, and `arena-eval` each have an actor+judge
+functional test harness under their own `tests/` folder — a *runner*
+(shared, parametrized: `eval_pipeline/tests/run_eval_pipeline_skill_tests.py
+--skill-root <suite> --cases <suite>/tests/<suite>_test_cases.jsonl`), a
+suite-specific JSONL fixture, and a testing guide. `openjudge-core` and the
+isolated skills don't have this yet (see each suite's `README.md` under
+"Validating the skills themselves" for the ones that do).
+
+### `ref-hallucination-arena` — intentional duplication, not a bug
+
+`academic-eval/03-ref-hallucination-arena/SKILL.md` and
+`arena-eval/02-ref-hallucination-arena/SKILL.md` start as full copies of the
+same content and are **independently maintained** — no shared `references/`
+directory, no symlink, no sync script. This is a direct consequence of the
+independent-installability constraint: a skill can't rely on content that
+lives outside its own folder. The two copies are free to diverge (the
+academic copy can lean into "how trustworthy are this paper's citations",
+the arena copy into "how does this model compare to others"); duplicated
+content is an accepted cost, not a problem to solve.
+
+## Isolated skills (single-skill, no suite directory)
+
+These have no other skill in this repo referencing them, so they stay at
+`skills/<name>/SKILL.md` with no suite folder — the simplest possible
+grouping (1 skill = 1 domain).
+
+| Skill | What it covers |
+|---|---|
+| [`claude-authenticity/`](claude-authenticity/) | Detect whether an API endpoint is genuinely backed by Claude (vs. a wrapper/proxy/impersonator); extract injected system prompts. Zero OpenJudge dependency. |
+| [`mmx-cli/`](mmx-cli/) | Generate text/image/video/speech/music via the MiniMax AI platform. Third-party CLI wrapper, zero OpenJudge dependency. |
+| [`find-skills-combo/`](find-skills-combo/) | Discover and recommend combinations of skills from the **external** open agent-skills ecosystem to cover a multi-part task. Unrelated to this repo's own suite grouping. |
+
+**Future rule**: if an isolated skill above ever becomes something a domain
+suite depends on or wraps, move the whole folder into that suite's directory
+(same copy-don't-link approach as `ref-hallucination-arena` above, or a
+rename+move if a single owner is clear) rather than keeping it independent
+and cross-linking from two places.
+
+## Breaking change: path migration (this reorg)
+
+This reorg moved 6 previously-published skills to new install paths. There
+are no redirect stubs — installing the old path will 404; use the new path.
+
+| Old path | New path |
+|---|---|
+| `skills/paper-review/` | `skills/academic-eval/01-paper-review/` |
+| `skills/bib-verify/` | `skills/academic-eval/02-bib-verify/` |
+| `skills/ref-hallucination-arena/` | `skills/academic-eval/03-ref-hallucination-arena/` **and** `skills/arena-eval/02-ref-hallucination-arena/` (two independent copies — see above) |
+| `skills/auto-arena/` | `skills/arena-eval/01-auto-arena/` |
+| `skills/openjudge/` | `skills/openjudge-core/01-graders-and-pipeline/` |
+| `skills/rl-reward/` | `skills/openjudge-core/02-rl-reward/` |
+
+The migrated skills' frontmatter `name` values also change to match their
+new directory names. Update explicit skill invocations to use the final
+component of each new path (for example, `paper-review` becomes
+`01-paper-review`, and `openjudge` becomes `01-graders-and-pipeline`). The
+two reference-arena copies use distinct names: `03-ref-hallucination-arena`
+in `academic-eval` and `02-ref-hallucination-arena` in `arena-eval`.
+
+`SkillLoader.load_from_directory("skills")` discovers skills recursively
+through suite directories. A directory containing `SKILL.md` is treated as
+one package, including its bundled references and examples.
+
+`skills/eval_pipeline/`, `skills/claude-authenticity/`, `skills/mmx-cli/`,
+and `skills/find-skills-combo/` are unaffected.
+
+## Adding a new skill
+
+- **Fits an existing suite's methodology or domain?** Add it as
+  `<suite>/<NN-name>/SKILL.md` with `name: NN-name` matching its directory,
+  choose a name unique across the repository, update that suite's `README.md`,
+  and add a router entry only if it creates real selection ambiguity with a
+  sibling (see each suite's README for its router-inclusion rationale).
+- **Genuinely stands alone?** Add it as `skills/<name>/SKILL.md` — no suite
+  folder needed.
+- **Either way**, the skill's `SKILL.md` must be installable on its own: only
+  `name` + `description` in frontmatter, no links outside its own folder (or
+  its own suite folder), and any non-`openjudge`/`rl-reward` pip dependency
+  listed explicitly under Prerequisites.

skills/academic-eval/00-academic-router/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: 00-academic-router
+description: >
+  Use when the user wants help with academic papers or citations but it's unclear
+  which specific workflow fits — reviewing a paper, checking a BibTeX file for fake
+  references, or benchmarking multiple LLMs on reference-recommendation accuracy.
+  Also use when the user mentions paper review, peer review, BibTeX verification,
+  citation checking, reference hallucination, or academic literature accuracy and
+  hasn't specified which of those three tasks they mean. This skill is the entry
+  router for the academic-eval suite: it asks one diagnostic question then routes
+  to the right sub-skill.
+---
+
+# Academic Eval Router
+
+Entry router for the `academic-eval` suite. You diagnose what the user actually
+wants and route them to one of three sub-skills. You don't review papers, verify
+BibTeX files, or run arena benchmarks yourself — you're the triage desk.
+
+Each sub-skill is self-contained: it carries inline everything it needs, so it can
+be installed and used on its own.
+
+## Diagnostic Question
+
+Ask (unless the user's request already makes the answer obvious):
+
+```
+To route you correctly, which of these matches what you want?
+
+a) Review a single paper (PDF or LaTeX source) for correctness/quality/novelty
+   — optionally also check its bibliography
+b) Check a standalone .bib file for fabricated or mismatched references
+   (no paper review needed)
+c) Benchmark/compare multiple LLMs on how often they hallucinate references
+   when asked to recommend citations (arena-style, many queries)
+```
+
+**Shortcut rule**: if the user already said "review my paper", "check this PDF",
+"verify this .bib file", or "compare models on reference hallucination", skip the
+question — the routing is already clear from their phrasing.
+
+## Triage Table
+
+| User says / has | Use workflow | What it does |
+|---|---|---|
+| "Review this paper" (PDF or `.tar.gz`/`.zip` TeX source) | `01-paper-review` | Multi-stage review: safety, correctness, quality/novelty score, criticality — optionally + BibTeX check |
+| "Review this paper AND check its references" | `01-paper-review` | Same pipeline with `--bib` set — one run covers both |
+| "Just check this .bib file, no paper" | `02-bib-verify` | Cross-checks every entry against CrossRef/arXiv/DBLP, flags `verified`/`suspect`/`not_found` |
+| "Compare N models on how often they cite fake papers" / "benchmark reference hallucination rate" | `03-ref-hallucination-arena` | Runs many recommendation queries per model, verifies every returned reference, ranks models by hallucination rate |
+| "Compare models on general quality/response, not specifically citations" | — | Not this suite — see the `arena-eval` suite's `01-auto-arena` instead |
+
+## Key distinctions
+
+- **`01-paper-review` vs `02-bib-verify`**: both use the same underlying
+  `cookbooks.paper_review` pipeline. Use `01-paper-review` whenever a paper file
+  exists (even if the *only* thing the user cares about is the bibliography —
+  `--bib_only` mode is documented there). Use `02-bib-verify` only when there is
+  **no paper**, just a loose `.bib` file to sanity-check.
+- **`01-paper-review`/`02-bib-verify` vs `03-ref-hallucination-arena`**: the first
+  two evaluate *one document's* existing references after the fact. The third
+  evaluates *model behavior* — how often a model invents fake citations when
+  asked to recommend some, across a benchmark of queries and models. If the user
+  wants a leaderboard/ranking of models, not a report on one document, route to
+  `03-ref-hallucination-arena`.
+
+## Output
+
+```
+Recommended workflow: `[skill-name]`
+
+Why: [one sentence tying the user's request to the triage table row]
+```
+
+Recommend exactly one workflow. If the request spans two (e.g., "review this
+paper, and separately benchmark 3 models on citation accuracy"), say so
+explicitly and give both, in the order the user would naturally do them.

skills/paper-review/SKILL.md → skills/academic-eval/01-paper-review/SKILL.md

@@ -1,5 +1,5 @@
 ---
-name: paper-review
+name: 01-paper-review
 description: >
   Review academic papers for correctness, quality, and novelty using OpenJudge's
   multi-stage pipeline. Supports PDF files and LaTeX source packages (.tar.gz/.zip).

skills/bib-verify/SKILL.md → skills/academic-eval/02-bib-verify/SKILL.md

@@ -1,5 +1,5 @@
 ---
-name: bib-verify
+name: 02-bib-verify
 description: >
   Verify a BibTeX file for hallucinated or fabricated references by cross-checking
   every entry against CrossRef, arXiv, and DBLP. Reports each reference as
@@ -73,5 +73,5 @@ Each reference entry is assigned one of three statuses:
 
 ## Additional resources
 
-- Full pipeline options: [../paper-review/reference.md](../paper-review/reference.md)
-- Combined PDF review + BibTeX verification: [../paper-review/SKILL.md](../paper-review/SKILL.md)
+- Full pipeline options: [../01-paper-review/reference.md](../01-paper-review/reference.md)
+- Combined PDF review + BibTeX verification: [../01-paper-review/SKILL.md](../01-paper-review/SKILL.md)

skills/ref-hallucination-arena/SKILL.md → skills/academic-eval/03-ref-hallucination-arena/SKILL.md

@@ -1,5 +1,5 @@
 ---
-name: ref-hallucination-arena
+name: 03-ref-hallucination-arena
 description: >
   Benchmark LLM reference recommendation capabilities by verifying every cited
   paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate,
@@ -254,7 +254,7 @@ evaluation_results/ref_hallucination_arena/
 
 ## Additional resources
 
-- Full config examples: [cookbooks/ref_hallucination_arena/examples/](../../cookbooks/ref_hallucination_arena/examples/)
-- Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../docs/validating_graders/ref_hallucination_arena.md)
+- Full config examples: [cookbooks/ref_hallucination_arena/examples/](../../../cookbooks/ref_hallucination_arena/examples/)
+- Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../../docs/validating_graders/ref_hallucination_arena.md)
 - Official dataset: [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena)
 - Leaderboard: [openjudge.me/leaderboard](https://openjudge.me/leaderboard)

skills/academic-eval/README.md

@@ -0,0 +1,63 @@
+# Academic Eval — paper review & citation accuracy
+
+A set of skills for academic-paper workflows built on OpenJudge: reviewing a
+paper end-to-end, spot-checking a bibliography for fabricated references, or
+benchmarking how often LLMs hallucinate citations at scale.
+
+Each skill is a self-contained workflow in `<NN-name>/SKILL.md`. Start at
+`00-academic-router` if you're not sure which one you need.
+
+## The workflows
+
+| # | Skill | Use it when |
+|---|---|---|
+| 00 | `00-academic-router` | You're not sure whether you want a paper review, a BibTeX check, or an arena-style benchmark. |
+| 01 | `01-paper-review` | You have a paper (PDF or LaTeX source) and want a multi-stage review — safety, correctness, quality/novelty, criticality, optionally + BibTeX. |
+| 02 | `02-bib-verify` | You have a standalone `.bib` file (no paper) and want to check it for fabricated/mismatched references. |
+| 03 | `03-ref-hallucination-arena` | You want to benchmark/compare multiple LLMs on how often they invent fake citations, across many queries. |
+
+## Relationship between the workflows
+
+`01-paper-review` and `02-bib-verify` both run on top of the same
+`cookbooks.paper_review` pipeline — `01` is the full document review (with an
+optional `--bib` flag to also verify references), `02` is the BibTeX-only mode
+for when there's no paper to review, just a bibliography to sanity-check.
+
+`03-ref-hallucination-arena` is a different axis entirely: instead of
+evaluating one document's existing references, it evaluates *model behavior* —
+how often a model fabricates references when asked to recommend citations,
+scored across a benchmark of queries and ranked across models. It shares
+`ref-hallucination-arena`'s content with the `arena-eval` suite's
+`02-ref-hallucination-arena` — the two copies are independently maintained
+(see the root [`skills/README.md`](../README.md) for why).
+
+## Dependencies
+
+| Skill | Cookbook |
+|---|---|
+| `01-paper-review` | `cookbooks/paper_review/` |
+| `02-bib-verify` | `cookbooks/paper_review/` (BibTeX-only mode) |
+| `03-ref-hallucination-arena` | `cookbooks/ref_hallucination_arena/` |
+
+```bash
+pip install py-openjudge litellm
+pip install matplotlib   # only needed by 03-ref-hallucination-arena (charts)
+```
+
+## Self-contained skills
+
+Each `<NN-name>/SKILL.md` is self-contained per the Anthropic Agent Skill
+protocol — it can be installed and used on its own without this README or the
+rest of the suite. Cross-references between skills in this suite use relative
+links scoped to this directory (e.g. `02-bib-verify` links to
+`../01-paper-review/`); there are no links out to other suites.
+
+## Validating the skills themselves
+
+This suite has an actor+judge functional test harness, same pattern as
+`eval_pipeline`: an *actor* model follows a skill to answer a realistic user
+request, and a separate *judge* model grades the answer against the case's
+acceptance criteria. See [`tests/academic_eval_testing_guide.md`](tests/academic_eval_testing_guide.md)
+for how to run it — it reuses `eval_pipeline`'s runner (parametrized, not
+duplicated) against this suite's own 12 test cases in
+[`tests/academic_eval_test_cases.jsonl`](tests/academic_eval_test_cases.jsonl).