perf(index): filter find-links entries before url conversion

[KRRT7]

Summary

• Classify local --find-links directory entries by filename before converting them to file URLs
• Preserve the same file links and HTML page candidates for valid package archives and HTML pages
• Avoid URL conversion for unrelated files in large flat directories

Performance Model

Observed path	Target path
Convert every directory entry path to a URL before checking whether it is usable	Check the entry name first, then convert only retained entries
Run URL quoting and mimetypes.guess_type() on unrelated files	Keep unrelated files on the cheap discard path
Broad scan pays conversion cost for files that are never exposed as links	Broad scan only pays conversion cost for package files and HTML pages

Profile before the change confirmed the cost: in 20 scans of a directory with 5,000 unrelated files, 50 wheels, and 10 HTML pages, _scan_directory() spent about 1.98s in path_to_url() and about 1.01s in HTML MIME checks.

Benchmark

Apple M3, 24 GiB RAM, CPython 3.14.5

Target workload: _FlatDirectoryToUrls.project_name_to_urls scanning a local --find-links directory with:

• 5,000 unrelated .txt files with spaces/symbols in their names
• 50 valid demo_pkg wheel filenames
• 10 HTML page files

Expected output is unchanged: 50 URLs under demo-pkg and 10 page candidate URLs.

	Min	Median	Mean	OPS	Rounds
6b0011b49 (base)	48.325ms	49.340ms	49.522ms	20.2 ops/s	40
9f8243c09 (head)	15.411ms	15.752ms	15.710ms	63.7 ops/s	40
Speedup	3.14x	3.13x	3.15x	3.15x

Reproduce the benchmark locally

uv run python -c 'import pathlib, statistics, tempfile, time
from pip._internal.index.sources import _FlatDirectoryToUrls

root = pathlib.Path(tempfile.mkdtemp(prefix="pip-find-links-scan-"))
for i in range(5000):
    (root / f"irrelevant file {i:04d} with spaces and symbols # ignored.txt").touch()
for i in range(50):
    (root / f"demo_pkg-{i}.0-py3-none-any.whl").touch()
for i in range(10):
    (root / f"page-{i}.html").touch()

def scan_once():
    source = _FlatDirectoryToUrls(str(root))
    urls = source.project_name_to_urls
    pages = source.page_candidates
    assert len(urls["demo-pkg"]) == 50
    assert len(pages) == 10

for _ in range(3):
    scan_once()
samples = []
for _ in range(40):
    start = time.perf_counter()
    scan_once()
    samples.append(time.perf_counter() - start)
print(f"min={min(samples)*1000:.3f}ms median={statistics.median(samples)*1000:.3f}ms mean={statistics.mean(samples)*1000:.3f}ms ops={1/statistics.mean(samples):.1f} rounds={len(samples)}")'

Changelog

Added news/d54d50c4-da3a-402b-886d-e6cecc11237a.bugfix.rst.

Stack

Order	PR	Branch	Merge after
1	Current PR	perf/find-links-scan-filter	main

Existing open performance PRs were checked and are independent of this change: #14122, #14108, #14106, #14088, #14103, #14045, #14026, and #13860.

Test plan

• Benchmarked base vs. head with the command above
• Profiled the baseline with cProfile
• uv run ruff check src/pip/_internal/index/sources.py tests/unit/test_collector.py
• uv run pytest tests/unit/test_collector.py -q