Research for some additional pytest plugins

[tomschr]

Situation

ℹ️ It's an idea. This does NOT mean we have to implement all below. If feasible, we can split this issue into several sub-issues.

I came across some pytest plugins which could be helpful to our existing test suite:

• pytest-datadir GitHub

  • What it does: It automatically locates and provides dedicated test data directories based on your test file structure.

  • Why use it: Instead of manually reading relative paths to your "real world" documentation sample fixtures, you can just use the datadir fixture. It copies your static test repository to a temporary directory for each test run so your actual assets remain pristine.

• pyfakefs (via pytest-pyfakefs) GitHub

  • What it does: Fakes the entire underlying Python file system in memory.

  • Why use it: If docbuild generates hundreds of HTML/JSON/XML files and you want the tests to run blazing fast without hitting actual disk I/O, this plugin simulates the file writing and reading natively.

• syrupy (Snapshot Testing) GitHub

  • What it does: Enables snapshot testing within pytest.

  • Why use it: When testing the actual terminal output (stdout/stderr) or the exact structure of a generated file, you can assert against a "snapshot" file. If the tool's verbose output formatting changes slightly in the future, you don't have to rewrite fifty test assertions; you just update the snapshot with a single CLI command (pytest --snapshot-update).

• subtests Doc

  • What id does: Not a plugin per se, but integrated pytest since version 9.0. Multiple small assertions or distinct logical checks are grouped into single tests. Can be used to dynamically created tests which are not known at collection time.
  • Why to use it: If an early assertion fails, the remaining assertions are skipped. Using subtests allows pytest to continue running the rest of the block and report multiple isolated failures, giving a better picture of what broke.

Use Case

Current state for data files

The test suite relies heavily on inline, multiline string literals to represent configuraiton files, XML data, and manifest structures. While there is a tests/data directory present, the dominant pattern is to inline and strings and use tmp_path to write them out manually.

• Readability: Moving large XML/TOML blocks out of the test functions into dedicated files drastically reduces visual noise, allowing developers to focus on the test logic rather than scrolling past large data blobs.
• Maintainability: External data files benefit from IDE features like syntax highlighting, linting, and automatic formatting (especially useful for XML/RNC files).
• Reusability: The same data files can be easily reused across different test modules using the injected shared_datadir fixture.

Current state of "fake" filesystems

There is extensive use of the tmp_path fixture (100+ occurrences) combined with directory creation and file writing.

• Removes Boilerplate & Hacks: It completely eliminates the need for manual monkeypatch.setattr("builtins.open", ...) or StringIO setups. You can mock out a complex directory structure in one line.
• Simplifies Configuration Setup: Instead of continually modifying application settings to point to a temporary folder (e.g., ctx.envconfig.paths.config_dir = tmp_path / "config.d"), you can use pyfakefs to mock the default system paths like /etc/docbuild directly in memory.
• Speed: The virtual filesystem operates entirely in memory, which is faster than writing physical files via tmp_path.

Current state of "snapshot" testing

The tests contain numerous assertions against nested dictionary structures (e.g., in models/test_manifest.py) and CLI string outputs. For CLI outputs, the current pattern relies on substring matching (e.g., assert "sles" in result.output).

• Comprehensive CLI Validation: Replacing brittle substring matching with assert result.output == snapshot guarantees that entire command-line layouts, spacings, and formatting are regressions-tested. Substring checks often fail to catch broken table formatting or unexpected extra output.

• Reduces Test Size: Removes massive dictionary or list assertions from the test file itself.

• Easy Updates: When the CLI output or manifest structure intentionally changes, developers don't have to manually update complex assertions; they simply run pytest --snapshot-update.

Possible Implementation

• For pytest-datadir:

  • Locate massive inline XML strings.
  • Create a folder or use tests/data and save them as .xml files.
  • Inject the datadir or shared_datadir fixture to the test to load the files.

• For pyfakefs:

  • Find tests that mock open() or use the tmp_path fixture.
  • Add the fs fixture to the test.
  • Use fs.create_file() or fs.create_dir() to set up the environment, including system paths like /etc/docbuild directly, bypassing the need to dynamically alter application config paths to point to tmp_path.

• For syrupy:

  • Look for tests validating CLI output or complex JSON/dict structures.
  • Replace the manual checks with assert result == snapshot
  • Generate snapshots. Run pytest --snapshot-update. This creates a __snapshots__ directory containing the expected output. Review these files carefully and commit them to Git.

[sushant-suse]

Great research! These plugins could definitely clean up the test suite and make future test writing much faster.** Since this is an exploratory issue and replacing things like tmp_path across 100+ tests is a big architectural shift, I propose we test the waters first.

Here are my initial thoughts on the implementation plan once we agree to move forward:

1. pytest-datadir & syrupy: These seem like quick, low-risk, high-reward wins. We can immediately spin these out into their own sub-issues and start moving the massive inline XML/TOML strings into a tests/data folder, and updating the CLI tests to use snapshots.
2. pyfakefs: I love the idea of mocking /etc/docbuild directly in memory without hacking the config paths. However, since docbuild relies on external binaries (like git and daps), we will just need to verify in the PoC that pyfakefs doesn't block those subprocesses from accessing the real filesystem when they need to.
3. subtests (pytest 9.0): Since we are already on pytest 9.x, we can organically start adopting this pattern for any new tests or when refactoring older, multi-assertion tests.

Let me know what you think of the PoC approach.

[tomschr]

I completely agree. We can use a small part were we can integrate it and check if it works. If it gives us more pain than gain, that's not what we want. 😊

Perhaps we can split this into subtasks?