Disclaimer : sorry for the edits. I still think it could be usefull to add a clarification, as it seems passing the language argument appears to change word preprocessing at some step, especially for french characters.
In the tutorial, the language = XXX option is used even though precomputed embeddings are passed to the model.
As far as I initially understood, language is actually disabled and has no effect when an embedding model is explicitly specified.
https://github.com/MaartenGr/BERTopic/blob/f9697602d57fc6acb8ac304026ac3c9aecd8a031/bertopic/_bertopic.py#L168
language: The main language used in your documents. The default sentence-transformers
model for "english" is `all-MiniLM-L6-v2`. For a full overview of
supported languages see bertopic.backend.languages. Select
"multilingual" to load in the `paraphrase-multilingual-MiniLM-L12-v2`
sentence-transformers model that supports 50+ languages.
NOTE: This is not used if `embedding_model` is used.
EDIT: Oh, it does seems to change the handling of French-specific characters and spelling (for example in Name and Representation : e.g. é, ç, etc.).
-> I might be wrong but :
I initially fought it also had no effet when precomputed embeddings are passed to fit/fit_transform.
However, when precomputed embeddings are passed, no embedding_model is explicitly provided to the BERTopic constructor. Instead, the embedding step is simply bypassed later when calling fit_transform.
Therefore, here:
self.language = language if not embedding_model else None
still leaves self.language set to its default value, "english", when no embedding_model is explicitly provided, or to the one provided.
This matters because later (here), in _preprocess_text, BERTopic applies:
if self.language == "english":
cleaned_documents = [
re.sub(r"[^A-Za-z0-9 ]+", "", doc)
for doc in cleaned_documents
]
As a result, if precomputed embeddings are used without explicitly setting language="french" (or another appropriate language), French-specific characters can be removed before the vectorization step.
So perhaps the documentation could clarify that using precomputed embeddings bypasses the embedding-generation step, but does not disable language globally: the language setting can still affect later text preprocessing and topic representation.
A small footnote might help prevent users working with precomputed embeddings from accidentally applying English-specific preprocessing to non-English documents.
Disclaimer : sorry for the edits. I still think it could be usefull to add a clarification, as it seems passing the
languageargument appears to change word preprocessing at some step, especially for french characters.In the tutorial, the
language = XXXoption is used even though precomputed embeddings are passed to the model.As far as I initially understood, language is actually disabled and has no effect when an embedding model is explicitly specified.
https://github.com/MaartenGr/BERTopic/blob/f9697602d57fc6acb8ac304026ac3c9aecd8a031/bertopic/_bertopic.py#L168
EDIT: Oh, it does seems to change the handling of French-specific characters and spelling (for example in Name and Representation : e.g. é, ç, etc.).
-> I might be wrong but :
I initially fought it also had no effet when precomputed embeddings are passed to fit/fit_transform.
However, when precomputed embeddings are passed, no embedding_model is explicitly provided to the BERTopic constructor. Instead, the embedding step is simply bypassed later when calling fit_transform.
Therefore, here:
self.language = language if not embedding_model else Nonestill leaves self.language set to its default value, "english", when no embedding_model is explicitly provided, or to the one provided.
This matters because later (here), in _preprocess_text, BERTopic applies:
As a result, if precomputed embeddings are used without explicitly setting language="french" (or another appropriate language), French-specific characters can be removed before the vectorization step.
So perhaps the documentation could clarify that using precomputed embeddings bypasses the embedding-generation step, but does not disable language globally: the language setting can still affect later text preprocessing and topic representation.
A small footnote might help prevent users working with precomputed embeddings from accidentally applying English-specific preprocessing to non-English documents.