test: add golden outputs for docs.bzl scenarios

.github/workflows/_test.yml

@@ -47,8 +47,11 @@ jobs:
       - name: Prepare test report directory
         run: mkdir -p tests-report
 
-      - name: Run public docs.bzl integration tests
-        run: .venv_docs/bin/python -m pytest -vv src/tests/docs_bzl --junitxml=tests-report/docs_bzl.xml
+      - name: Run cacheable public docs.bzl tests
+        run: .venv_docs/bin/python -m pytest -vv -m bazel_cached src/tests/docs_bzl --junitxml=tests-report/docs_bzl_cached.xml
+
+      - name: Run slow public docs.bzl tests
+        run: .venv_docs/bin/python -m pytest -vv -m bazel_slow src/tests/docs_bzl --junitxml=tests-report/docs_bzl_slow.xml
 
       - name: Run bazel test targets
         run: bazel test --lockfile_mode=error //... --build_tests_only

.pre-commit-config.yaml

@@ -18,8 +18,12 @@ repos:
       - id: check-toml
       - id: check-json
         exclude: ^\.vscode/ # those are actually jsonc files
+      # Golden HTML is compared byte-for-byte, so preserve the exact generated
+      # whitespace and EOF instead of normalizing these fixture files.
       - id: end-of-file-fixer
+        exclude: ^src/tests/docs_bzl/scenarios/.*/_expected/.*\.html$
       - id: trailing-whitespace
+        exclude: ^src/tests/docs_bzl/scenarios/.*/_expected/.*\.html$
       - id: check-merge-conflict
       - id: check-case-conflict
       - id: mixed-line-ending

pyproject.toml

@@ -92,6 +92,8 @@ log_file_format = "[%(asctime)s.%(msecs)03d] [%(levelname)-3s] [%(name)s] %(mess
 log_file_date_format = "%Y-%m-%d %H:%M:%S"
 
 markers = [
+    "bazel_cached: successful build-only test expected to reuse Bazel actions",
+    "bazel_slow: runtime execution or an uncached expected Bazel failure",
     "metadata",
     "test_properties(dict): Add custom properties to test XML output",
 ]

src/tests/docs_bzl/README.md

@@ -48,3 +48,66 @@ Run via:
 
 The suite is deliberately separate from `bazel test //...`, since pytest is its
 driver. CI stores its JUnit XML together with the Bazel test reports.
+
+Tests are marked according to whether their Bazel work can be reused from the
+action cache. To run only the cacheable cases:
+
+    .venv_docs/bin/python -m pytest -vv -m bazel_cached src/tests/docs_bzl
+
+To run the runtime and expected-failure cases:
+
+    .venv_docs/bin/python -m pytest -vv -m bazel_slow src/tests/docs_bzl
+
+The CI workflow runs these two commands in that order, so the cacheable tests
+provide the fast first feedback before the runtime tests start.
+
+The default command above runs both groups. Golden scenarios are marked
+automatically: scenarios with a `docs` expected output are `bazel_slow` cases;
+scenarios containing only successful build outputs are `bazel_cached` cases.
+Expected-failure tests are also `bazel_slow`, because failed analysis does not
+produce reusable action outputs.
+
+## Expected target outputs
+
+Positive scenarios may check in direct output files below an `_expected`
+directory next to their fixture. An `_expected/<target>` directory contains
+expected files relative to that target's output root; an
+`_expected/<target>.<suffix>` file checks one file output. Only files present
+below an expected directory are compared, so unrelated Bazel or Sphinx output
+is ignored.
+
+For example:
+
+```text
+scenarios/basic_docs/
+└── _expected/
+    ├── docs/
+    │   └── index.html
+    ├── needs_json/
+    │   └── needs.json
+    └── generated_config.py
+```
+
+The short target names are mapped to their real Bazel labels and output roots in
+`expected_outputs.py`; this also includes internal generated targets when their
+output is part of the contract. JSON files are parsed and compared as
+deterministically formatted, sorted JSON so that their checked-in form remains
+readable. Other files, including HTML, are compared byte-for-byte; expected
+files should therefore contain only deterministic output. Missing expected files
+or changed content fail the test, while additional actual files do not.
+
+To refresh the checked-in files after an intentional output change, run the
+updater for one scenario:
+
+    .venv_docs/bin/python -m src.tests.docs_bzl.expected_outputs --update basic_docs
+
+The scenario argument uses the same path as pytest, for example
+`reference_integration/modern_module`. Omitting it updates all scenarios:
+
+    .venv_docs/bin/python -m src.tests.docs_bzl.expected_outputs --update
+
+The updater only overwrites files that already exist below `_expected`; it does
+not add every generated file or remove files. Review the resulting Git diff and
+run the normal pytest suite afterwards. When updating all scenarios, their
+build-only targets are grouped into one Bazel invocation; runtime targets still
+run one at a time.