mw2slob

This is a tool to convert MediaWiki content to slob dictionaries from Wikimedia Enterprise HTML Dumps or from CouchDB instances created by mwscrape.

Using HTML dumps is recommended for most users. mwscrape downloads rendered articles via MediaWiki API and can be used to obtain articles HTML for namespaces that are not available as enterprise HTML dumps.

Installation

mw2slob requires Python 3.9 or newer and depends on the following components:

• Slob (not published on PyPI - installed straight from GitHub, see below)
• couchdb-python
• lxml
• cssselect
• cssutils
• BeautifulSoup
• PyICU (via Slob), which needs the ICU library itself

Two native dependencies need a system library present before Python packaging can build them: lxml needs libxml2~/~libxslt, and PyICU (a dependency of Slob) needs ICU. Install those first.

On Ubuntu/Debian:

sudo apt-get install libxml2-dev libxslt1-dev libicu-dev pkg-config

On macOS (Homebrew):

brew install icu4c pkg-config

Homebrew’s icu4c is keg-only (not linked onto PATH), so pkg-config won’t find it unless you point it there explicitly - set this in your shell profile, or export it just before running uv~/~pip commands below:

export PKG_CONFIG_PATH="$(brew --prefix icu4c)/lib/pkgconfig"

With uv (recommended)

uv resolves and builds the whole dependency graph in one step, including slob pulled directly from its GitHub repo (it isn’t on PyPI, so this used to be a manual step) - no separate “install slob first” dance.

To work on mw2slob itself, clone it and let uv create a dedicated virtual environment:

git clone https://github.com/itkach/mw2slob.git
cd mw2slob
uv sync
uv run mw2slob --help

(uv sync creates .venv in the project directory; uv run uses it automatically. Activate it directly with source .venv/bin/activate if you’d rather not prefix every command with uv run.)

To just run mw2slob without cloning anything, uv can build and run it straight from GitHub into a disposable, cached environment. This always picks up the latest commit on the default branch:

uvx --from git+https://github.com/itkach/mw2slob mw2slob --help

To run a specific released version instead, name its tag in the URL:

uvx --from git+https://github.com/itkach/mw2slob@1.2 mw2slob --help

If mw2slob --version ever looks out of date right after a new tag is published - uv’s cache doesn’t always notice a tag added to a commit it already has - clear it once and try again:

uv cache clean

With plain pip

Create a virtual environment, then install mw2slob and slob (from GitHub, since it isn’t on PyPI) into it:

python3 -m venv .venv
source .venv/bin/activate
pip install git+https://github.com/itkach/mw2slob.git

Usage

With WE HTML Dumps

# get site's metadata ("siteinfo")
mw2slob siteinfo http://en.wiktionary.org > enwikt.si.json
# compile dictionary
mw2slob dump --siteinfo enwikt.si.json ./enwiktionary-NS0-20220120-ENTERPRISE-HTML.json.tar.gz -f wikt common

Note -f wikt common argument that specifies content filters to use when compiling this dictionary. Content filter is a text file containing list of CSS selectors (one per line). HTML elements matching these selectors will be removed during compilation. `mw2slob` includes several filters (see ./mw2slob/filters) that work well for most wikipedias and wiktionaries.

Wikimedia Enterprise HTML Dumps are available only for some namespaces. For most wikipedias the main namespace 0 - articles - is typically the only one of interest to most users. Wiktionaries, on the other hand, often make use of other such namespaces. For example, in English Wiktionary many articles include links to articles from Wiktionary or Appendix namespaces, so it makes sense to include their content into compiled dictionary and make these links internal dictionary links rather than link to Wiktionary web site.

These namespaces are not available as html dumps, but can be obtained via Mediawiki API via mwscrape. Let’s say we want to compile English Wiktionary and include the following namespaces in addition to the main articles: Appendix, Wiktionary, Rhymes, Reconstruction and Thesaurus (sampling random articles indicates that these namespaces are often referenced).

First, we examine siteinfo (saved in enwikt.si.json) and find that ids for these namespaces are:

Wiktionary	4
Appendix	100
Rhymes	106
Thesaurus	110
Reconstruction	118

Then we download rendered articles for these namespaces with mwscrape:

mwscrape https://en.wiktionary.org --db enwikt-wiktionary --namespace 4
mwscrape https://en.wiktionary.org --db enwikt-appendix --namespace 100
mwscrape https://en.wiktionary.org --db enwikt-rhymes --namespace 106
mwscrape https://en.wiktionary.org --db enwikt-thesaurus --namespace 110
mwscrape https://en.wiktionary.org --db enwikt-reconstruction --namespace 118

Each takes some time, but these are relatively small and don’t take too long.

Finally, compile the dictionary:

mw2slob dump --siteinfo enwikt.si.json \
        ./enwiktionary-NS0-20220420-ENTERPRISE-HTML.json.tar.gz \
        http://localhost:5984/enwikt-wiktionary \
        http://localhost:5984/enwikt-appendix \
        http://localhost:5984/enwikt-rhymes \
        http://localhost:5984/enwikt-thesaurus \
        http://localhost:5984/enwikt-reconstruction \
        -f wikt common --local-namespace 4 100 106 110 118

Note that `mw2slob dump` takes CouchDB URLs of the databases we created with mwscrape in addition to the dump file name.

Also note the `–local-namespace` parameter. This tells the compiler to make the links for these namespaces internal dictionary links, just like cross-article links, otherwise they would be converted to web links.

See mw2slob dump --help for complete list of options.

With mwscrape database

Assuming CouchDB server runs at localhost on port 5984 and has mwscrape database created with mwscrape simple.wikipedia.org and named simple-wikipedia-org, to create a slob using common and wiki content filters:

mw2slob scrape http://127.0.0.1:5984/simple-wikipedia-org -f common wiki

See mw2slob scrape --help for complete list of options