Add pyrefly to test_typecheck port/board matrix as xfail

[Copilot]

pyrefly support existed in the typecheck harness (typecheck.py, typecheck_pyrefly.py) but was never wired into the parametrized test_typecheck matrix that runs pyright/mypy/ruff across every stub version/port/board/feature combination.

Changes

• tests/quality_tests/test_snippets.py: added pyrefly to the linter parametrization for test_typecheck, marked xfail(strict=False) since pyrefly support is experimental — mirrors the pattern already used for the commented-out basilisk entry.

@pytest.mark.parametrize(
    "linter",
    [
        "pyright",
        "mypy",
        "ruff",
        pytest.param(
            "pyrefly",
            marks=pytest.mark.xfail(reason="pyrefly support is experimental", strict=False),
        ),
    ],
)

This exercises pyrefly against the full port/board/feature snippet matrix without failing CI on known gaps.

[Josverl]

Also add the astral ty typechecker to the quality tests and the just recipe, and wire it into the test matrix as xfail

[Josverl]

@copilot

Also add the astral ty typechecker to the quality tests and the just recipe, and wire it into the test matrix as xfail

[Copilot]

Also add the astral ty typechecker to the quality tests and the just recipe, and wire it into the test matrix as xfail

Added in 1b3b0b7 (see follow-up commit): new typecheck_ty.py module wired into typecheck.py, ty added to the install-linters/test-linter just recipes, and added to the test_typecheck linter matrix in test_snippets.py marked xfail (experimental), mirroring the pyrefly entry.

@copilot

Also add the astral ty typechecker to the quality tests and the just recipe, and wire it into the test matrix as xfail

Added in 7b74c4e: new typecheck_ty.py module wired into typecheck.py, ty added to the install-linters/test-linter just recipes and test extras, and added to the test_typecheck linter matrix in test_snippets.py marked xfail (experimental), mirroring the pyrefly entry. (Note: my earlier reply on this thread cited an incorrect commit hash — 7b74c4e is the correct one.)

[Josverl]

@copilot , I see that the test_typecheck_stdlib_only() test do not yet incluse pyrefly nor ty .
Please add them to that part as well

Or better - define the list of linters and their options once in a module, constant ( or ficture) and re-use that across test and test suites

[Copilot]

@copilot , I see that the test_typecheck_stdlib_only() test do not yet incluse pyrefly nor ty .
Please add them to that part as well

...

Done in c.f. this commit: added LINTER_PARAMS as a shared constant in typecheck.py (pyright, mypy, ruff, xfail pyrefly, xfail ty) and reused it in both test_snippets.py::test_typecheck and test_stdlib_only.py::test_typecheck_stdlib_only, so the stdlib-only suite now also runs pyrefly and ty (plus ruff).

[Copilot]

Pull request overview

Extends stub quality testing with experimental Pyrefly and ty support.

Changes:

• Adds Pyrefly and ty adapters and configuration.
• Adds both tools to the snippet matrix as non-strict xfails.
• Adds a dedicated Pyrefly test and documentation.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/quality_tests/typecheck.py	Dispatches Pyrefly and ty checks.
tests/quality_tests/typecheck_ty.py	Converts ty results to the shared format.
tests/quality_tests/typecheck_pyrefly.py	Converts Pyrefly results to the shared format.
tests/quality_tests/test_snippets.py	Adds experimental matrix entries.
tests/quality_tests/test_pyrefly.py	Adds dedicated Pyrefly coverage.
tests/quality_tests/readme.md	Documents the new checkers.
tests/quality_tests/feat_pyrefly/check_basics_01.py	Provides a Pyrefly test snippet.
tests/quality_tests/_configs/pyproject.toml	Configures Pyrefly and ty.
pyproject.toml	Adds test dependencies.
justfile	Installs and selects the new checkers.

Suppressed comments (2)

tests/quality_tests/typecheck_pyrefly.py:133

• An unexpected Pyrefly exit is converted to [], which is indistinguishable from a successful run with no diagnostics. pyrefly_to_pyright then creates a zero-error report, so the matrix records an allowed XPASS and the dedicated suite passes even though Pyrefly never completed. Propagate execution and JSON-parsing failures instead; the xfail marker will still absorb them in the experimental matrix while the dedicated test can detect a broken adapter.

            if result.returncode not in (0, 1):
                log.error(f"Pyrefly failed with returncode {result.returncode}: {result.stderr}")
                return []

tests/quality_tests/test_snippets.py:176

• The title and description scope this PR to wiring in Pyrefly, but this also adds ty to the full port/board/feature matrix (plus a new adapter, dependency, config, docs, and install changes). That materially expands runtime and maintenance scope without being described. Either remove the ty additions from this PR or update the PR metadata and validation plan to cover both checkers.

    pytestconfig: pytest.Config,
):
    if not snip_path_fx or not snip_path_fx.exists():
        pytest.skip(f"no feature folder for {feature}")
    caplog.set_level(logging.INFO)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.