Improve Ghost ingest language detection - #3
Merged
Conversation
Pure formatting ahead of the language-detection work, so that the functional diff which follows is readable rather than buried in reflowed decorators and call sites. black at its default line length (88) plus import sorting; no behaviour change. Verified by comparing the AST of each module before and after, ignoring import order.
maquchizi
force-pushed
the
improve-ghost-ingest-language-detection
branch
from
August 10, 2026 10:25
88c0f81 to
69934bb
Compare
Superdesk needs an ISO 639-1 code on every ingested item, and PesaCheck
publishes in English, Kiswahili, French, Somali, Afaan Oromo and
Amharic. Add a self-contained detector with two strategies:
- detect_language_from_tags: recent posts carry an explicit language tag
("English", "Afaan Oromo", ...). This is editorial metadata, so it is
authoritative and tried first. Matching is exact on the normalised
name or slug so a language is never confused with the country tag
beside it, which PesaCheck always sets: French/France,
Somali/Somalia. Where two language tags appear the lowest sort_order
wins, matching Ghost's primary tag.
- detect_language_from_text: for posts predating that convention.
Ethiopic script share identifies Amharic; the Latin-script languages
are separated by calibrated function-word coverage.
The classifier is deliberately closed over just these six languages
rather than delegating to a general-purpose detector. langdetect has no
model for Afaan Oromo and labels it Somali (both Cushitic, both using
the doubled-vowel Latin orthography), and it raises outright on
Amharic. Scoring only what we publish removes the failure mode where an
article is lost to a seventh language that was never an option.
Marker lists and the per-language coverage constants were built and
measured against the 50-post reference export in ops/local/ingest/ghost.
Bare "Oromo" is deliberately not accepted as a language tag: it is an
ethnonym first and a plausible topic tag on an English article, and
Oromo prose is recognised well by the text path, so an unmatched tag
degrades safely.
The module imports only the standard library. It deliberately lives in
pesacheck/ rather than pesacheck/ingest/: the ingest package's __init__
eagerly imports the parsers, which pull in superdesk.io and the native
libmagic dependency, so placing it there would mean this pure-stdlib
logic -- and its tests -- could not be imported without the whole
Superdesk stack.
Replace _guess_language, which only ever returned am, fr or en. Somali and Afaan Oromo both collapsed to English: langdetect was consulted but its result was trusted only for en/fr, and it has no Oromo model regardless. Measured against the 50-post reference export that was 92% overall but 0% on Somali and Oromo. _parse_language now resolves in three steps: Ghost's locale (normalised, so en-US becomes en), then the post's language tag, then the new text classifier. Tag records carry their slug so either the display name or the slug can name the language. Detection runs on Ghost's plaintext rendering rather than body_html, so markup no longer dilutes the word counts, and falls back to html when a post has no plaintext. Also coerce a null sort_order on tags: the existing sort would have raised TypeError comparing None to int, and sort_order now decides which language tag wins. Both paths reach 100% on the reference export -- via tags, and via text with the language tag stripped to simulate the older untagged posts.
langdetect was added as a direct requirement solely for Ghost ingest language detection, which no longer uses it. Nothing else in the tree imports it. six stays in requirements.txt: it is still required by flask-oidc-ex, oauth2client and python-dateutil.
maquchizi
force-pushed
the
improve-ghost-ingest-language-detection
branch
from
August 10, 2026 11:08
695c5be to
53108d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
publishes in.
_guess_languageonly ever returnedam,froren, so Somaliand Afaan Oromo both landed as English — 92% accurate overall on the 50-post
reference export, but 0% on those two. Kiswahili was never detected at all.
English,Afaan Oromo,...), which is editorial metadata and therefore authoritative. Posts predating
that tagging convention fall back to a text classifier closed over just these six
languages: Ethiopic script share for Amharic, calibrated function-word coverage
for the Latin-script five.
langdetect, which hasno Afaan Oromo model and labels it Somali (both Cushitic, both using the
doubled-vowel Latin orthography) and raises outright on Amharic.
langdetectisdropped as a dependency.
scripts/,settings.py,medium_parser.py, and others). Verified formatting-only — ASTidentical before and after for every file. Worth reviewing separately from, or
splitting out of, the ingest change.
Test plan
six languages, and the cases where it declines to guess.
ops/local/ingest/ghost) via the tagpath, and 100% again with the language tag stripped to exercise the text path.
French/France,Somali/Somalia,English/Ethiopia.Amharic; embedded English quotes don't flip Somali or Oromo articles.