Skip to content

fix(citations): honor the errs tolerance cap in CitationMixin fuzzy span matching - #2463

Closed
lntutor wants to merge 1 commit into
567-labs:mainfrom
lntutor:fix/citation-error-tolerance-off-by-one
Closed

fix(citations): honor the errs tolerance cap in CitationMixin fuzzy span matching#2463
lntutor wants to merge 1 commit into
567-labs:mainfrom
lntutor:fix/citation-error-tolerance-off-by-one

Conversation

@lntutor

@lntutor lntutor commented Jul 19, 2026

Copy link
Copy Markdown

What's broken

CitationMixin._get_span escalates the allowed regex edit distance in a loop whose guard is evaluated before the increment:

errs_ = 0
s = regex.search(f"({minor}){{e<={errs_}}}", major)
while s is None and errs_ <= errs:   # errs defaults to 5
    errs_ += 1
    s = regex.search(f"({minor}){{e<={errs_}}}", major)

When errs_ == errs (5) the body runs once more, bumping errs_ to 6 and searching with {e<=6}. So the effective fuzzy tolerance is errs + 1 = 6 — a hallucinated substring_quote that is 6 edits from any real source span is accepted and silently rewritten to that unrelated context substring, instead of being dropped as "not found":

class Answer(CitationMixin): pass
Answer.model_validate({"substring_quotes": ["0123ZZZZZZ"]}, context={"context": "0123456789"}).substring_quotes
# before: ['0123456789']   (6 edits away -- should be [])

This weakens the citation verification the mixin exists to provide.

Fix

Change the guard to errs_ < errs so the final search runs at exactly {e<=errs}. Quotes within errs edits still match.

Tests

Adds tests/dsl/test_citation.py (within-tolerance matches; beyond-tolerance dropped). The existing citation coverage in tests/coverage/test_dsl_small_coverage.py — which relies on a legitimate 1-edit match — stays green (16). Distinct from #2430 (regex-escape crash), which leaves the loop bound untouched.

…atching

_get_span escalates the allowed regex edit distance in a loop whose
guard (errs_ <= errs) is checked before the increment, so it runs one
extra search at e<=errs+1. A substring_quote up to 6 edits from any
source span (one beyond the default errs=5 cap) is therefore accepted
and silently rewritten to that unrelated context substring, instead of
being dropped as not-found -- weakening the citation verification the
mixin exists to provide.

Change the guard to errs_ < errs so the final search runs at exactly
e<=errs; quotes within errs edits still match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N6RtoHuxrDqTUo9Mw9h4Cv
@jxnl

jxnl commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Consolidated and shipped in #2495. Closing this focused patch as superseded; thank you for the contribution.

@jxnl jxnl closed this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants