Repository navigation
fix(indexes): file accented titles under their base letter - #118
Merged
Merged
Conversation
letter_bucket() sent every title not starting with ASCII a-z to '0-9', including accented Latin (Émile, über, Ñandú), so French/German/Spanish title indexes collapsed into one catch-all page. NFKD-fold the first character to its base letter in both the Python bucketer (static mode) and _TITLE_BUCKET_SQL (dynamic mode), and add a test that pins the two together; there was none, so they could drift silently. Non-Latin scripts and letters without a decomposition (ß, Æ, Ø) still go to '0-9'. No expression index uses the SQL, so no migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MjiSEYcZ2q8F7bke96VGHT
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MjiSEYcZ2q8F7bke96VGHT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The title index sent every title not starting with ASCII a–z to
0-9, including accented Latin. SoÉmile,über,Ñandúandİstanbulall landed in the catch-all. On a French, German or Spanish subreddit, much of the title index was one0-9page.Change
The first character is NFKD-folded to its base letter before bucketing, in both places that bucket:
html_modules/html_static_indexes.pyletter_bucket(): static mode (Pythonunicodedata)core/postgres_database.py_TITLE_BUCKET_SQL: dynamic mode (PostgreSQLnormalize(…, NFKD), built in, no extension)Bucket names are unchanged (
a–z,0-9), so URLs are unchanged. Re-exported archives just re-file those titles. No expression index uses the SQL, so there is no migration.Tests
_TITLE_BUCKET_SQLagainst 19 titles and compares withletter_bucket(). I confirmed it fails when only the Python side changes (Émile Zola: '0-9' != 'e').letter_bucketunit tests. The oldtest_non_ascii_goes_to_catchallassertedÉmile → 0-9; that is exactly the behavior this PR changes.Also marks the item done in the Unicode roadmap spec.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MjiSEYcZ2q8F7bke96VGHT