diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..7e7ba3d --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,30 @@ +# Every path needs an owner that GitHub can actually resolve. +# +# A handle that does not resolve makes the whole rule silently inert: GitHub does +# not warn, it just never requests the review. PLAN.md lists @pratik for +# submissions/**, but that is a placeholder, not a real handle. It is deliberately +# omitted until confirmed, and added at the Phase 12 transfer alongside the +# required-review rule. +# +# Note also that on a personal-account repository a code owner must have write +# access, and teams cannot be granted access at all. So @org/team entries would +# be inert here even if written correctly. They become usable after transfer to +# drexel-ice, not before. + +* @JiwaniZakir + +# This is the untrusted-input boundary. Everything under here arrives from +# outside the lab: checkpoints are pickles and predict.py is arbitrary Python. +/submissions/ @JiwaniZakir + +# The workflows are the executable surface of this repository. A change here can +# turn a read-only validation job into one holding a write token, which is +# exactly the failure mode CODEOWNERS exists to slow down. +/.github/ @JiwaniZakir + +# TODO(phase-12): once real handles are confirmed, restore these AND enable +# require_code_owner_reviews with required_approving_review_count: 1. Until both +# are on, CODEOWNERS is advisory only. +# +# /submissions/ @pratik-handle +# /.github/ @JiwaniZakir @second-reviewer diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..d53a5b9 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,9 @@ +blank_issues_enabled: false +contact_links: + - name: Report a security vulnerability + url: https://github.com/JiwaniZakir/eda-schema-experiments/security/advisories/new + about: Please report privately rather than in a public issue. + + - name: Something wrong with the leaderboard site + url: https://github.com/JiwaniZakir/eda-schema-leaderboard/issues/new/choose + about: Wrong numbers, broken pages and rendering issues belong in the site repo. diff --git a/.github/ISSUE_TEMPLATE/ingest-failure.yml b/.github/ISSUE_TEMPLATE/ingest-failure.yml new file mode 100644 index 0000000..ad1e052 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/ingest-failure.yml @@ -0,0 +1,53 @@ +name: My submission fails to ingest +description: Derivation broke on a folder that follows the documented layout +title: "[ingest]: " +labels: ["parser"] +body: + - type: markdown + attributes: + value: | + **This is our bug, not yours.** + + Nothing is asked of submitters. Folders are ingested exactly as they come + off the cluster, and every field is derived. If derivation fails on your + folder, the parser needs fixing - please do not add a metadata file to + work around it, because that defeats the point. + + - type: input + id: path + attributes: + label: Path to the folder + placeholder: submissions/your-name/total_area_prediction/... + validations: + required: true + + - type: textarea + id: error + attributes: + label: What CI reported + description: Paste the failing job output. It will be formatted as a code block. + render: shell + validations: + required: true + + - type: dropdown + id: where + attributes: + label: Which check failed + options: + - schema + - size + - Both + - Neither, the numbers came out wrong + validations: + required: true + + - type: textarea + id: layout + attributes: + label: Directory layout + description: >- + The output of `find -maxdepth 4 -type d`. Stage names + contain underscores and PDK directory names are uppercase, and most + parse failures come down to one of those two. + render: shell diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..3d86c1a --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,17 @@ +version: 2 +updates: + # Actions only. This repository holds data and submissions, not a Python + # project - there is no pyproject.toml or lockfile here, and the validator it + # runs comes from the site repo, which has its own Dependabot config. + # + # This matters for the SHA-pinned workflows: pinning is only safe to maintain + # because Dependabot updates pins. Without this file the pins would rot. + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + commit-message: + prefix: chore + groups: + actions: + patterns: ["*"] diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..81ed0ca --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,34 @@ + + +## What this changes + +## Submission checklist + +- [ ] The folder is exactly as it came off the cluster, with nothing added +- [ ] No file exceeds 1 MB (CI blocks it; checkpoints are around 32 KB) +- [ ] No tfevents (`events.out.tfevents*`) - the targets in them are z-scored +- [ ] No PNGs - these belong in a Release and are referenced by URL +- [ ] Directory names follow the documented layout, with uppercase PDK names + +## Division + +- [ ] **Open** - any training regime +- [ ] **Closed** - the canonical leave-one-circuit-out split, canonical features + and target + +If you are unsure, leave both unchecked. Division is derived from the tree shape +rather than declared, and a per-circuit training set resolves to Open +automatically. You cannot misclassify yourself into Closed. + +## For maintainers + +- [ ] `schema` and `size` both green, and `schema` reported a non-zero submission count +- [ ] Nothing under `.github/` changed, or the change was reviewed as a + permissions change diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000..200271c --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,72 @@ +name: codeql + +on: + pull_request: + push: + branches: [main] + schedule: + - cron: "0 7 * * 1" + +permissions: {} + +# Actions are pinned to commit SHAs because the analyze job holds +# `security-events: write`. Same rule as the site repo: any workflow with a write +# scope is pinned, so a retagged upstream cannot reach it. + +jobs: + # CodeQL hard-errors with "could not process any code written in Python" when a + # language has no source, and this repository holds data, not code - there is + # no Python here until a submission arrives carrying predict.py. + # + # This was not theoretical. The first version of this file listed + # [actions, python] unconditionally and `analyze (python)` failed on its very + # first run with exit code 32. Detecting instead means the Python leg starts by + # itself the moment a submission lands, rather than sitting red until someone + # remembers to re-enable it - or worse, being deleted because it was noisy, on + # the one repository where untrusted Python is the whole point. + # + # `actions` always applies: the workflows themselves are this repo's executable + # surface. + detect: + runs-on: ubuntu-latest + permissions: + contents: read + outputs: + languages: ${{ steps.pick.outputs.languages }} + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + persist-credentials: false + + - id: pick + run: | + langs='["actions"]' + if [ -n "$(git ls-files '*.py')" ]; then + langs='["actions","python"]' + fi + echo "languages=$langs" >> "$GITHUB_OUTPUT" + echo "analysing: $langs" + + analyze: + needs: detect + runs-on: ubuntu-latest + permissions: + contents: read + security-events: write + strategy: + fail-fast: false + matrix: + language: ${{ fromJSON(needs.detect.outputs.languages) }} + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + persist-credentials: false + + - uses: github/codeql-action/init@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3 + with: + languages: ${{ matrix.language }} + queries: security-extended + + - uses: github/codeql-action/analyze@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3 + with: + category: /language:${{ matrix.language }} diff --git a/.github/workflows/notify-site.yml b/.github/workflows/notify-site.yml index da9e289..b4575f3 100644 --- a/.github/workflows/notify-site.yml +++ b/.github/workflows/notify-site.yml @@ -6,25 +6,45 @@ on: permissions: {} +# This workflow has never actually dispatched. SITE_DISPATCH_TOKEN was unset, so +# it printed a notice and exited 0 - reporting success on every push to main +# while doing nothing, which left the site's repository_dispatch path entirely +# untested behind a green tick. +# +# Now a repository VARIABLE gates the job. `secrets` is not available in a +# job-level `if:` but `vars` is, so when dispatch is not turned on the job does +# not run and the workflow reports SKIPPED rather than success. When it IS turned +# on and the credential is missing, that is a hard failure, because at that point +# a silent no-op is a broken deploy chain rather than an unconfigured one. +# +# TO ENABLE: +# 1. Create a fine-grained PAT scoped to JiwaniZakir/eda-schema-leaderboard +# only, with Contents: read and write. Nothing else. +# 2. Store it as the secret SITE_DISPATCH_TOKEN. +# 3. Set the repository variable SITE_DISPATCH_ENABLED to "true". +# +# This is the one sanctioned stored credential in either repository. GITHUB_TOKEN +# cannot dispatch across repositories, and the workload identity federation used +# for the Claude action covers the Anthropic side only, not GitHub's. + jobs: dispatch: + if: vars.SITE_DISPATCH_ENABLED == 'true' runs-on: ubuntu-latest permissions: {} steps: - name: Tell the site to rebuild env: - # A fine-grained PAT with contents:write on the site repo only. - # GITHUB_TOKEN cannot dispatch across repositories. - # - # No payload is sent. The site's deploy workflow deliberately ignores - # client_payload and always builds its own main, so nothing here can - # influence what gets published. TOKEN: ${{ secrets.SITE_DISPATCH_TOKEN }} run: | if [ -z "$TOKEN" ]; then - echo "::notice::SITE_DISPATCH_TOKEN is not set; skipping rebuild trigger." - exit 0 + echo "::error::SITE_DISPATCH_ENABLED is true but SITE_DISPATCH_TOKEN is not set" + exit 1 fi + + # No payload is sent. The site's deploy workflow deliberately ignores + # client_payload and always builds its own main, so nothing here can + # influence what gets published. curl -sSf -X POST \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer $TOKEN" \ diff --git a/.github/workflows/validate-submission.yml b/.github/workflows/validate-submission.yml index 1ca1e90..0ec6d9f 100644 --- a/.github/workflows/validate-submission.yml +++ b/.github/workflows/validate-submission.yml @@ -23,6 +23,9 @@ jobs: steps: - uses: actions/checkout@v4 with: + # Full history so the base commit is present and the diff below can be + # computed locally rather than trusting an event field for it. + fetch-depth: 0 persist-credentials: false - uses: astral-sh/setup-uv@v5 @@ -36,11 +39,76 @@ jobs: path: .site persist-credentials: false + # Whether the pull request touched submissions/ decides whether an empty + # scan is acceptable. Computed here because only the workflow knows what + # the PR changed; the validator cannot infer it. + # + # BASE_SHA comes from the event payload and is a hex SHA produced by + # GitHub, but it is passed through env rather than interpolated into the + # script, per the rule that no event field is ever spliced into a run:. + - id: touched + env: + BASE_SHA: ${{ github.event.pull_request.base.sha }} + run: | + # Only record extensions count, matching tools/submissions.py SUFFIXES. + # A plain `-- submissions/` pathspec counts anything, including this + # directory's own README, and then --require-nonempty fires on a docs + # change that adds no records - which it promptly did on the first run. + # The trigger has to ask the same question the scanner answers. + changed=$(git diff --name-only "$BASE_SHA"...HEAD -- \ + 'submissions/**/*.json' 'submissions/**/*.yaml' 'submissions/**/*.yml' \ + | wc -l) + echo "count=$changed" >> "$GITHUB_OUTPUT" + echo "the pull request changes $changed submission record(s)" + - name: JSON Schema and guard layers 1 to 5 working-directory: .site + env: + # Absolute, because the validator runs from .site and a relative path + # would resolve against the wrong tree - which is the bug this whole + # step replaces. + SUBMISSIONS: ${{ github.workspace }}/submissions + REQUIRE: ${{ steps.touched.outputs.count != '0' && '--require-nonempty' || '' }} run: | - uv sync --all-extras - uv run eda-validate + # This previously ran `uv run eda-validate` with no argument, from + # inside .site. That validated the LEADERBOARD's own registries and + # baseline, reported success, and never opened the submission at all. + # A submission could have contained anything and this check would have + # been green. + # + # --submissions points it at the actual submission tree, and it prints + # the number of records it examined. --require-nonempty makes an empty + # scan a failure when the PR did touch submissions/, so "found nothing" + # can no longer be mistaken for "found nothing wrong". + # + # Plain `uv sync`, NOT --all-extras. The site's `ingest` extra pulls + # torch and the whole CUDA stack - about 2.5 GB, measured - which this + # job has no use for. It is also the wrong instinct on the one runner + # that handles untrusted submissions: every package installed here is + # attack surface, and none of it is needed to validate JSON. + uv sync + + set +e + out=$(uv run eda-validate --submissions "$SUBMISSIONS" $REQUIRE 2>&1) + status=$? + set -e + printf '%s\n' "$out" + + # A guard on the guard. `eda-validate` took no arguments until the + # --submissions flag landed, and argparse was not in the picture - so an + # older validator does not reject the flag, it silently ignores it and + # exits 0. That is exactly what happened on this workflow's first run: + # `validate: 3 checks, 0 failures`, no submission count, green tick, + # submission never opened. + # + # The count line is the proof the flag was honoured. Its absence means + # we are validating nothing, whatever the exit status says. + if ! printf '%s' "$out" | grep -q "submissions from"; then + echo "::error::the validator did not report a submission count; it ignored --submissions and examined nothing" + exit 1 + fi + + exit $status size: runs-on: ubuntu-latest @@ -58,8 +126,10 @@ jobs: # -z because git quotes non-ASCII paths, and a quoted path silently # fails [ -f ] and gets skipped. fail=0 + seen=0 limit=$((MAX_FILE_MB * 1024 * 1024)) while IFS= read -r -d '' f; do + seen=$((seen + 1)) [ -f "$f" ] || continue case "$f" in *events.out.tfevents*) @@ -72,4 +142,11 @@ jobs: fail=1 fi done < <(git ls-files -z) + echo "checked $seen tracked files against the ${MAX_FILE_MB} MB cap" + + # A scan that walked zero files is a broken scan, not a clean repo. + if [ "$seen" -eq 0 ]; then + echo "::error::no tracked files were scanned; the guard did not run" + exit 1 + fi exit $fail diff --git a/submissions/README.md b/submissions/README.md new file mode 100644 index 0000000..7a76520 --- /dev/null +++ b/submissions/README.md @@ -0,0 +1,50 @@ +# Submissions + +One directory per submission: `submissions//`. + +This directory is committed rather than left empty on purpose. CI validates +submissions by pointing the leaderboard's validator at this path, and a missing +directory is reported as a failure rather than as "nothing to check" - so the +directory existing is what keeps the guard honest before the first submission +lands. + +## What to send + +The folder exactly as it came off the cluster. Nothing added. + +Every leaderboard field is derived - from the directory path, `eval.log`, +`run.log`, `hparams.yaml`, one checkpoint per architecture, and `git log`. There +is no metadata file to fill in, and adding one will not help. If derivation fails +on your folder, that is a parser bug on our side: open an issue with the +"My submission fails to ingest" template. + +Two fields are genuinely underivable. `seed` and raw predictions emit `null` and +set the badges `reproducibility: unverified` and `plots: static`. Neither blocks +a submission. + +## What CI enforces + +- **`schema`** validates every `.json`, `.yaml` and `.yml` record under this + directory against the submission schema, and reports how many it examined. A + pull request that touches `submissions/` and yields zero records fails, because + "found nothing" must not read as "found nothing wrong". +- **`size`** rejects any file over 1 MB and any `events.out.tfevents*`. + +Checkpoints are around 32 KB and belong here. PNGs are roughly 380 KB, are +regenerable, and belong in a Release referenced by URL. + +## Two things that bite + +**Stage names contain underscores**, so `global_place` and `detailed_place` do +not survive splitting a directory name on `_`. Parsing is anchored against the +registry vocabulary instead. + +**PDK directory names are uppercase** - `default_config_ASAP7_cts`, not `asap7`. +Registry IDs are lowercase and parsing normalizes to them. + +## Divisions + +Division follows from the shape of your tree and is never declared. A +single-circuit-sized training set per leaf directory resolves to `open`, because +the canonical Closed split is leave-one-circuit-out. You cannot misclassify +yourself into Closed.