fix(indexes): file accented titles under their base letter

CHANGELOG.md

@@ -16,6 +16,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   (CodeQL `py/url-redirection`).
 
 ### Fixed
+- Title index filed every title not starting with ASCII a–z under `0-9`,
+  including accented Latin (`Émile`, `über`, `Ñandú`). The first character is
+  now NFKD-folded to its base letter in both static (Python) and dynamic (SQL)
+  modes, and a parity test keeps the two in sync. Non-Latin scripts still use
+  `0-9` (#118).
 - Comment/post import dropped an entire COPY batch (up to ~1,000 rows) when the
   source contained a duplicate ID within that batch — the staging table's
   PRIMARY KEY aborted the whole COPY, not just the dup. Observed on the Voat

core/postgres_database.py

@@ -1387,11 +1387,10 @@ def stream_post_titles(
             return
 
     # First-character bucket of a title, matching letter_bucket() in
-    # html_modules/html_static_indexes.py: single lowercase a-z, else '0-9'
-    # (a multi-char lowercase expansion like 'İ' -> 'i̇' fails the regex).
-    _TITLE_BUCKET_SQL = (
-        "CASE WHEN lower(substr(ltrim(title), 1, 1)) ~ '^[a-z]$' THEN lower(substr(ltrim(title), 1, 1)) ELSE '0-9' END"
-    )
+    # html_modules/html_static_indexes.py: NFKD-fold the first character to its
+    # base letter ('É' -> 'e'), then single lowercase a-z, else '0-9'.
+    _TITLE_BUCKET_FIRST = "lower(substr(normalize(substr(ltrim(title), 1, 1), NFKD), 1, 1))"
+    _TITLE_BUCKET_SQL = f"CASE WHEN {_TITLE_BUCKET_FIRST} ~ '^[a-z]$' THEN {_TITLE_BUCKET_FIRST} ELSE '0-9' END"
 
     def get_title_letter_counts(self, subreddit: str) -> dict[str, int]:
         """Per-letter title counts for a subreddit ('a'..'z' + '0-9'), for dynamic title browsing."""

html_modules/html_static_indexes.py

@@ -3,6 +3,7 @@
 
 import math
 import re
+import unicodedata
 from typing import Any
 
 from html_modules.html_url import generate_domain_display_and_hover
@@ -21,13 +22,19 @@
 
 
 def letter_bucket(title: str | None) -> str:
-    """Bucket a title by its first character: 'a'..'z', or '0-9' for everything else."""
+    """Bucket a title by its first character: 'a'..'z', or '0-9' for everything else.
+
+    Accented and compatibility forms of Latin letters fold to their base letter
+    ('É' → 'e', 'ü' → 'u', fullwidth A → 'a'), so French/German/Spanish titles
+    land under their letter. Mirrored in SQL by PostgresDatabase._TITLE_BUCKET_SQL.
+    """
     stripped = (title or "").strip()
     if not stripped:
         return "0-9"
-    # .lower() can expand one char to several (e.g. 'İ' → 'i' + combining dot),
-    # and such strings compare inside 'a'..'z' lexicographically — require len 1.
-    first = stripped[0].lower()
+    # NFKD splits 'É' into 'E' + combining accent; keep the base character.
+    # .lower() can still expand one char to several, and such strings compare
+    # inside 'a'..'z' lexicographically — require len 1.
+    first = unicodedata.normalize("NFKD", stripped[0])[0].lower()
     return first if len(first) == 1 and "a" <= first <= "z" else "0-9"
 
 

roadmap/05-unicode-foreign-language-support.md

@@ -5,7 +5,7 @@
 
 **Goal:** Ensure redd-archiver correctly stores, searches, displays, and indexes content from non-English subreddits, including CJK (Chinese/Japanese/Korean), Cyrillic, Arabic, and other scripts.
 
-**Problem:** The codebase stores and displays Unicode text correctly (UTF-8 throughout). The original English-only FTS and CJK truncation issues are fixed. What remains: CJK text is not tokenized by FTS (Phase 2), pg_trgm cannot see non-ASCII characters on the deployed `ctype=C`, and the title index sends every non-ASCII-initial title (including accented Latin) to the `0-9` bucket.
+**Problem:** The codebase stores and displays Unicode text correctly (UTF-8 throughout). The original English-only FTS and CJK truncation issues are fixed. What remains: CJK text is not tokenized by FTS (Phase 2), pg_trgm cannot see non-ASCII characters on the deployed `ctype=C`, and the title index still sends non-Latin-initial titles (CJK, Cyrillic, Arabic) to the `0-9` bucket.
 
 ---
 
@@ -69,7 +69,7 @@ Arctic Shift API search fields relevant to us: `title`, `selftext`, `query` (bot
 | CJK not tokenized by FTS | `'simple'` parser | A space-less CJK run is one lexeme: `to_tsvector('simple', '日本語の検索')` → `'日本語の検索'`. A search for `日本語` does not match it. Phase 2 target. | **High** for CJK archives |
 | ~~Search operator regex `\w+`~~ | `utils/search_operators.py` | ✅ **Not a Unicode problem.** Python 3 `\w` is Unicode-aware; `sub:русский` and `author:Müller` parse (verified 2026-09-29). The real bug was hyphens: `author:some-user` was cut at `-`. Fixed in #115, which also fixed uppercase operators being left in the query text. | — |
 | ~~Smart text truncation~~ | `html_modules/jinja_filters.py` `truncate_smart` | ✅ **Fixed.** Hard-truncates space-less (CJK) runs instead of a no-op word break. | — |
-| Title index bucketing (Feature 1) | `html_modules/html_static_indexes.py` `letter_bucket` | **Built, but everything outside ASCII a–z goes to `0-9`**, including accented Latin (`Éclair`, `über`), Cyrillic, and CJK. A non-English subreddit's title index is one giant `0-9` page. | **Medium** |
+| Title index bucketing (Feature 1) | `html_static_indexes.py` `letter_bucket` + `PostgresDatabase._TITLE_BUCKET_SQL` | ✅ **Accented Latin fixed (#118):** NFKD folding puts `Éclair` under `e` and `über` under `u`, in both static (Python) and dynamic (SQL) modes, with a parity test. Non-Latin scripts (CJK, Cyrillic, Arabic) and `ß`/`Æ`/`Ø` still go to `0-9`. | **Low** (non-Latin only) |
 | Flair slugs for non-Latin flair | `html_static_indexes.py` `flair_slugs` | Non-alphanumeric-ASCII flair collapses to `flair`, `flair-2`, … (unique, but unreadable URLs). | **Low** |
 
 ### Investigated (needs re-verification on Alpine)
@@ -206,11 +206,11 @@ def truncate_smart(text, length=150, suffix="..."):
 
 **File:** `html_modules/jinja_filters.py:116-137`
 
-### 4. Title index bucketing (Medium — built without this)
+### 4. Title index bucketing (Low — accented Latin fixed; non-Latin remains)
 
 **Current (2026-09-29):** Feature 1 shipped `letter_bucket()` in `html_modules/html_static_indexes.py`, which sends every title whose first character is not ASCII `a`–`z` to `0-9`. That includes accented Latin (`Éclair`, `über`), so French/German/Spanish subreddits are affected, not just non-Latin ones.
 
-**Cheap first step:** fold diacritics before bucketing (`unicodedata.normalize("NFKD", c)[0]`), so `É` → `e` and `ü` → `u`.
+**Cheap first step:** ✅ Done in #118. Diacritics fold before bucketing (NFKD in Python and in SQL), so `É` → `e` and `ü` → `u`. Remaining: the non-Latin bucket below.
 
 **Proposed approach:** Add a catch-all bucket for titles starting with non-Latin characters.
 

tests/test_static_indexes.py

@@ -35,13 +35,64 @@ def test_empty_and_none(self):
         assert letter_bucket("   ") == "0-9"
         assert letter_bucket(None) == "0-9"
 
-    def test_non_ascii_goes_to_catchall(self):
-        assert letter_bucket("Émile Zola") == "0-9"
+    def test_accented_latin_folds_to_base_letter(self):
+        assert letter_bucket("Émile Zola") == "e"
+        assert letter_bucket("über alles") == "u"
+        assert letter_bucket("Ñandú") == "n"
+        assert letter_bucket("Çava") == "c"
+        assert letter_bucket("İstanbul news") == "i"  # 'İ' → 'I' + combining dot
+
+    def test_compatibility_forms_fold(self):
+        assert letter_bucket("\uff21pple fullwidth") == "a"  # FULLWIDTH LATIN CAPITAL A
+        assert letter_bucket("final ligature") == "f"
+
+    def test_non_latin_goes_to_catchall(self):
         assert letter_bucket("日本語タイトル") == "0-9"
-
-    def test_multichar_lowercase_expansion(self):
-        # 'İ'.lower() expands to 'i' + combining dot — must not leak as a bucket
-        assert letter_bucket("İstanbul news") == "0-9"
+        assert letter_bucket("Привет мир") == "0-9"
+        assert letter_bucket("مرحبا") == "0-9"
+
+    def test_latin_letters_without_decomposition_go_to_catchall(self):
+        # No NFKD decomposition to a-z: stay in the catch-all
+        assert letter_bucket("ßtraße") == "0-9"
+        assert letter_bucket("Æsir") == "0-9"
+        assert letter_bucket("Øresund") == "0-9"
+
+
+_BUCKET_PARITY_TITLES = [
+    "Apple pie",
+    "zebra",
+    "  Mid spaces",
+    "9 things",
+    '"Quoted title"',
+    "[deleted]",
+    "",
+    "Émile Zola",
+    "über alles",
+    "Ñandú",
+    "İstanbul news",
+    "\uff21pple fullwidth",  # FULLWIDTH LATIN CAPITAL A
+    "final ligature",
+    "日本語タイトル",
+    "Привет мир",
+    "مرحبا",
+    "ßtraße",
+    "Æsir",
+    "Øresund",
+]
+
+
+class TestTitleBucketSqlParity:
+    """Dynamic mode buckets in SQL; static mode in Python. They must agree."""
+
+    def test_sql_bucket_matches_letter_bucket(self, postgres_db):
+        with postgres_db.pool.get_connection() as conn, conn.cursor() as cur:
+            for title in _BUCKET_PARITY_TITLES:
+                cur.execute(
+                    # Interpolates a class constant, not input; the title is a bound parameter
+                    f"SELECT {postgres_db._TITLE_BUCKET_SQL} AS bucket FROM (VALUES (%s::text)) AS t(title)",  # noqa: S608
+                    (title,),
+                )
+                assert cur.fetchone()["bucket"] == letter_bucket(title), title
 
 
 @pytest.mark.unit