Add MTEB(slk, v1) results for Qwen/Qwen3-Embedding-4B

[andrejridzik]

Results for Qwen/Qwen3-Embedding-4B on MTEB(slk, v1)

This includes:

• results for 20 new tasks in MTEB(slk, v1)
• results for "slk" language for 5 multilingual tasks that already had results for some languages
• results for all languages in 3 multilingual tasks where there was a dataset revision update (existent conflict between code and results in MTEB)
  • Tasks: BelebeleRetrieval, FloresBitextMining, NTREXBitextMining
  • Note: Dataset revision has changed, so slightly different scores are expected

[andrejridzik]

Note: I wasn't sure how to deal with the date and evaluation_time for the 5 updated result files, so I just replaced them with my new values as this is how it was done for the previous language (tha-Thai) as well.

[github-actions]

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: Qwen/Qwen3-Embedding-4B

Results for Qwen/Qwen3-Embedding-4B

task_name	Qwen/Qwen3-Embedding-4B	google/gemini-embedding-001	intfloat/multilingual-e5-large	Max result	Model with max result	In Training Data
BelebeleRetrieval	.805	.907	.779	.985	sionic-ai/comsat-embed-ko-8b-preview	False
DemagogSKNLI	.870	nan	.844	.896	intfloat/multilingual-e5-large-instruct	False
FloresBitextMining	.740	.837	.811	.982	google/gemini-embedding-2-preview	False
MultiEupSlovakGenderClassification	.752	nan	.611	.628	intfloat/multilingual-e5-large-instruct	False
MultiEupSlovakPartyClassification	.460	nan	.479	.498	nomic-ai/nomic-embed-text-v2-moe	False
MultilingualSentimentClassification	.775	nan	.890	.897	Snowflake/snowflake-arctic-embed-l-v2.0	False
NTREXBitextMining	.876	.936	.914	.988	google/gemini-embedding-2-preview	False
OpusSlovakEnglishBitextMining	.841	nan	.862	.891	sentence-transformers/LaBSE	False
PravdaSKTagClustering	.499	nan	.612	.614	intfloat/multilingual-e5-base	False
PravdaSKURLClustering	.611	nan	.532	.604	intfloat/multilingual-e5-large-instruct	False
SIB200Classification	.823	nan	.730	.969	codefuse-ai/F2LLM-v2-8B	False
SKQuadRetrieval	.648	nan	.702	.710	nomic-ai/nomic-embed-text-v2-moe	False
SMESumCategoryClustering	.468	nan	.275	.442	intfloat/multilingual-e5-large-instruct	False
SMESumRetrieval	.848	nan	.859	.883	jinaai/jina-embeddings-v3	False
SkQuadReranking	.648	nan	.728	.728	intfloat/multilingual-e5-large	False
SlovakHateSpeechClassification.v2	.597	nan	.571	.599	intfloat/multilingual-e5-large-instruct	False
SlovakMovieReviewSentimentClassification.v2	.833	nan	.707	.862	intfloat/multilingual-e5-large-instruct	False
SlovakNLI	.842	nan	.773	.842	intfloat/multilingual-e5-large-instruct	False
SlovakParlaSentClassification	.717	nan	.559	.588	sentence-transformers/paraphrase-multilingual-mpnet-base-v2	False
SlovakPharmacyDrMaxReranking	.932	nan	.937	.946	jinaai/jina-embeddings-v3	False
SlovakPharmacyMojaLekarenReranking	.967	nan	.974	.974	intfloat/multilingual-e5-large	False
SlovakSTS	.863	nan	.798	.858	jinaai/jina-embeddings-v3	False
SlovakSumRetrieval	.918	nan	.930	.939	jinaai/jina-embeddings-v3	False
SlovakSumSTS	.925	nan	.920	.943	intfloat/multilingual-e5-large-instruct	False
SlovakSumURLClustering	.298	nan	.200	.309	intfloat/multilingual-e5-large-instruct	False
WebFAQBitextMiningQAs	.991	nan	.991	.999	Snowflake/snowflake-arctic-embed-l-v2.0	False
WebFAQBitextMiningQuestions	.986	nan	.975	.992	Snowflake/snowflake-arctic-embed-l-v2.0	False
WebFAQRetrieval	.780	nan	.797	.865	codefuse-ai/F2LLM-v2-14B	False
Average	.761	.894	.741	.801	nan	-

Model have high performance on these tasks: SlovakSTS,MultiEupSlovakGenderClassification,PravdaSKURLClustering,SlovakParlaSentClassification,SMESumCategoryClustering

---

[KennethEnevoldsen]

@andrejridzik can I ask you to also submit the run_settings.jsonl? it seems like there is score disagreements from previous runs

[andrejridzik]

@andrejridzik can I ask you to also submit the run_settings.jsonl? it seems like there is score disagreements from previous runs

@KennethEnevoldsen As I mentioned in the description, dataset revision has changed for three tasks, so slightly different scores were expected. We can see it in the failing tests on NTREXBitextMining, BelebeleRetrieval and FloresBitextMining.

I have just modified the batch size, but rest should be standard. All three look similar to this:

{"task": "NTREXBitextMining", "split": "test", "subset": "afr_Latn-dan_Latn", "version": {"mteb": "2.16.0", "torch": "2.11.0", "sentence-transformers": "5.5.1", "flash-attn": "2.8.3.post1", "transformers": "5.10.2"}, "encode_kwargs": {"batch_size": 8}}