Trustworthy LilyPond reading; release 0.3.0

.github/workflows/ci.yml

@@ -2,9 +2,9 @@ name: CI
 
 on:
   push:
-    branches: [master]
+    branches: [master, dev]
   pull_request:
-    branches: [master]
+    branches: [master, dev]
 
 env:
   CARGO_TERM_COLOR: always
@@ -64,6 +64,32 @@ jobs:
       - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends lilypond
       - run: cargo test --test lilypond_oracle -- --ignored --nocapture
 
+  lilypond-corpus:
+    # The grammar against LilyPond's own regression tests and snippets:
+    # valid files the parser flags with syntax errors (tests/ly_corpus.rs).
+    name: LilyPond corpus
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: true
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - name: Fetch LilyPond v2.26.0 (regression tests and snippets only)
+        run: |
+          git clone --depth 1 --branch v2.26.0 --filter=blob:none --sparse \
+            https://gitlab.com/lilypond/lilypond.git "$RUNNER_TEMP/lilypond"
+          git -C "$RUNNER_TEMP/lilypond" sparse-checkout set input/regression Documentation/snippets
+      - run: cargo test --test ly_corpus -- --ignored --nocapture
+        env:
+          LYTK_LILYPOND_SRC: ${{ runner.temp }}/lilypond
+      # The LilyPond-shaped fuzz properties, with many more cases than in
+      # `cargo test` (release build: inputs at the reader's bounds are slow
+      # unoptimised).
+      - run: cargo test --release --test fuzz_inputs lilypond_shaped
+        env:
+          LYTK_FUZZ_CASES: "1024"
+
   msrv:
     name: MSRV (rust-version in Cargo.toml)
     runs-on: ubuntu-latest
@@ -78,8 +104,13 @@ jobs:
       - run: cargo check --all-targets
 
   python:
-    name: Python tests
+    name: Python ${{ matrix.python }} tests
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # The abi3 floor (lilycorpus runs on 3.10) up to the newest.
+        python: ["3.10", "3.11", "3.12", "3.13"]
     steps:
       - uses: actions/checkout@v4
         with:
@@ -88,7 +119,7 @@ jobs:
       - uses: Swatinem/rust-cache@v2
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.12"
+          python-version: ${{ matrix.python }}
       - name: Create and expose virtualenv
         run: |
           python -m venv .venv
@@ -98,11 +129,14 @@ jobs:
       - run: python -m pip install --upgrade pip
       - run: pip install maturin pytest pytest-benchmark scipy
       # CPU-only torch so the torch dataset-adapter tests run instead of skipping
-      # (scipy above enables the eval-metric tests too).
-      - run: pip install torch --index-url https://download.pytorch.org/whl/cpu
+      # (scipy above enables the eval-metric tests too). On one Python only:
+      # the frameworks are large, and the others test the bindings.
+      - if: matrix.python == '3.12'
+        run: pip install torch --index-url https://download.pytorch.org/whl/cpu
       # TensorFlow too: its dtype rules are stricter than torch's, so the tf
       # data-loader path fails on inputs the torch path accepts. Without this
       # every tf test silently skips and that class of bug ships.
-      - run: pip install tensorflow-cpu
+      - if: matrix.python == '3.12'
+        run: pip install tensorflow-cpu
       - run: maturin develop
       - run: pytest tests/ -v

.github/workflows/release.yml

@@ -79,6 +79,16 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+      # A tag other than the version would publish a wheel under the wrong
+      # number (the version comes from Cargo.toml).
+      - name: Tag matches the version
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          version=$(sed -n 's/^version = "\(.*\)"$/\1/p' Cargo.toml | head -n 1)
+          if [ "${GITHUB_REF_NAME#v}" != "$version" ]; then
+            echo "::error::tag $GITHUB_REF_NAME does not match version $version in Cargo.toml"
+            exit 1
+          fi
       - uses: dtolnay/rust-toolchain@stable
       - uses: Swatinem/rust-cache@v2
       - run: cargo test

CLAUDE.md

@@ -55,7 +55,7 @@ Six layers:
    - `TimeSignature.beats` is a `String` (supports compound like "3+2"), use `.beats_fraction()` for the `Frac` value
 
 3. **Adapters** (`src/adapters/`) — format converters, all go through IR:
-   - `ly_to_ir/` — LilyPond parser using tree-sitter AST walk. Handles `\relative`, variables, `<< \\ >>` multi-voice, `\include`, figured bass, lyrics. The walk writes a positioned `Timeline` per part (`ly_to_ir/timeline.rs`); measures are made once, at score assembly
+   - `ly_to_ir/` — LilyPond parser using tree-sitter AST walk. Handles `\relative`, variables, `<< \\ >>` multi-voice, figured bass, lyrics; it does not follow `\include` (it warns: flatten first). The walk writes a positioned `Timeline` per part (`ly_to_ir/timeline.rs`); measures are made once, at score assembly. It reports `Diagnostic`s (`src/diagnostics.rs`): syntax errors from the tree (`ly_to_ir/syntax.rs`) and, from the walk's drop sites (`WalkState::report`), what it rejects or does not read. `builtins.rs` (LilyPond's commands, for `unknown-command`) is generated by `scripts/ly_builtins.py`. Strings, markup text and `\header` fields are decoded in `ly_to_ir/text.rs`, as LilyPond's lexer does
    - `mxml_to_ir/` — MusicXML reader using `musicxml` crate (typed struct traversal). Handles `.xml` and `.mxl` natively.
    - `ir_to_ly/` (~3400 lines) — IR to LilyPond emitter. Handles multi-staff piano scores, voice filtering, relative pitch mode
    - `ir_to_mxml/` — IR to MusicXML writer using `musicxml` crate (struct construction + serialization). Native MXL support.
@@ -71,7 +71,7 @@ Six layers:
 
 5. **ML representations** (`src/representations/`) — muspy-style encodings over the Layer-1 Music tree: `note_array` (onset/duration/pitch/velocity rows), `event_sequence` (Performance-RNN-style event codes), `piano_roll` (T × 128 matrix), `metrics`. All exposed to Python (`to_note_array`, `to_piano_roll`, `to_event_sequence` + inverses). Structured note navigation lives in `src/navigation.rs`.
 
-6. **CLI** (`src/lytk/cli.py`) — the one `lytk` command, a Typer app over the Python bindings: `convert`, `transpose` (`-s`/`--interval`/`--to-key`), `invert`, `retrograde`, `change-language`, `abs2rel`, `rel2abs`, `info`, `positions`, `bundle`, `batch`, `diff`, `flatten`. Folder conversion and `batch` run in worker processes (`ProcessPoolExecutor`). There is no Rust binary: the crate is a library only. The Python package `src/lytk/` also ships datasets (`lytk.datasets`) and eval metrics (`lytk.metrics`).
+6. **CLI** (`src/lytk/cli.py`) — the one `lytk` command, a Typer app over the Python bindings: `convert`, `transpose` (`-s`/`--interval`/`--to-key`), `invert`, `retrograde`, `change-language`, `abs2rel`, `rel2abs`, `info`, `positions`, `bundle`, `batch`, `diff`, `flatten`, `check`. Folder conversion and `batch` run in worker processes (`ProcessPoolExecutor`). There is no Rust binary: the crate is a library only. The Python package `src/lytk/` also ships datasets (`lytk.datasets`) and eval metrics (`lytk.metrics`).
 
 ## Key Design Patterns
 

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lytk"
-version = "0.2.0"
+version = "0.3.0"
 edition = "2021"
 # Floor set by our oldest dependency (clap 4.6). Matters because the PyPI sdist
 # is compiled on the user's own toolchain.

README.md

@@ -58,6 +58,20 @@ lytk.to_lilypond(score, "output.ly")         # or to_musicxml, to_midi, to_abc,
 abc = lytk.to_abc(score)                     # without a path, writers return the text
 ```
 
+Check LilyPond before you rely on it:
+
+```python
+for d in lytk.check_lilypond(text, semantic=True):   # errors and warnings, in source order
+    print(d)                                         # 3:12: error: missing `}` [missing-token]
+score = lytk.from_lilypond("score.ly", strict=True)  # raises lytk.LilyPondSyntaxError on an error
+score.diagnostics                                    # what the reading reported, either way
+score.header                                         # every \header field, as a dict
+fields = lytk.header_fields(text)                    # …or read from the text: key, value, span
+```
+
+A reader raises `lytk.ParseError` (a `ValueError`) when its input cannot be
+read, and `OSError` when the file cannot be opened.
+
 Turn a score into arrays:
 
 ```python
@@ -101,6 +115,7 @@ lytk convert input.xml -o output.ly           # formats are taken from the exten
 lytk convert corpus/ -o out/ -f xml -j 8      # a whole folder, in parallel
 lytk transpose input.ly -s 3 -o up.ly
 lytk flatten score.ly -o flat.ly              # inline every \include
+lytk check score.ly --semantic                # report the errors of LilyPond files
 lytk info input.mxl
 ```
 
@@ -126,9 +141,10 @@ lyrics); slurs and ornaments are not performed.
 
 ## Not there yet
 
-- Tablature, percussion (unpitched notes) and fretboard diagrams.
-- MIDI import transcribes the file as it is. It does not yet quantize, separate
-  voices, detect tuplets or split the hands of a piano part.
+- Tablature, unpitched notes and fretboard diagrams (LilyPond drums are read
+  as their General MIDI keys on a percussion staff).
+- Performed MIDI is quantized less reliably when its tempo is far from the
+  file's own, when chords are released unevenly, or at a low resolution.
 - Non-traditional key signatures, cross-staff notes (`\change Staff`), and
   Humdrum spine splits (`*^`, `*v`). Files that use spine splits are rejected
   with an error.

SECURITY.md

@@ -19,9 +19,10 @@ reachable from a malicious input file are in scope, in particular:
 
 - panics or unbounded memory/CPU use while parsing (a reachable
   denial-of-service for any service that converts user uploads)
-- path traversal or unintended file reads — note that LilyPond `\include` is
-  followed by design, so callers that accept untrusted `.ly` files should run
-  conversion in a sandbox and pass only trusted `-I` search paths
+- path traversal or unintended file reads. The LilyPond reader does not
+  follow `\include` (it reports it); `lytk flatten` / `lytk.flatten` does, by
+  design, so callers that flatten untrusted `.ly` files should run it in a
+  sandbox and pass only trusted `-I` search paths
 - decompression bombs in `.mxl` archives (these are read with a size cap)
 
 Release builds keep `overflow-checks` on, so an integer overflow panics rather