A model can top the MTEB average and be ordinary at retrieval

A team picks an embedding model by one number

An embedding model turns a piece of text into a vector: a list of numbers that places the text as a point in space. A search system uses those points to pull back the passages nearest a question. MTEB, the Massive Text Embedding Benchmark, is the standard way of comparing those models. In the paper that defines it, MTEB runs each frozen model across 8 task types spread over 58 datasets in 112 languages, and the authors report results for 33 models.¹

Two things share the name and are worth keeping apart. The benchmark is the collection of datasets and the rules for scoring a model on them. The leaderboard is the ranked table people actually read, a public page on Hugging Face that lists models best to worst.² The benchmark produces a score for each model. The leaderboard sorts on it. A team choosing an embedding model for a retrieval system lets that one score stand in for everything the model might do well or badly.

One average, eight different rulers

The tasks MTEB scores are not variations on one skill. Retrieval asks a model to rank the right passage above the wrong ones. Classification asks it to sort texts into labeled bins. Clustering asks it to group texts with no labels at all. Because the tasks differ, each is scored by its own metric on its own scale. Retrieval uses nDCG@10, classification uses accuracy, clustering uses v-measure, and so on through all eight.¹ These numbers do not share a unit. An nDCG@10 and an accuracy measure different things on different scales, and there is no natural exchange rate between them.

The overall score does the simplest possible thing with these unlike numbers. It averages each dataset's main-metric score across all the datasets, with no weighting, and the leaderboard sorts on that mean.¹ The headline English figure runs over 56 of the 58 datasets, dropping the two bitext-mining sets, which are multilingual. The shape is the same either way: a score on one dataset, folded in with every other, then divided by the count.

Folding scores this way treats a point of nDCG@10 on a retrieval set and a point of accuracy on a classification set as worth the same, because the average adds them and divides. Text embedding has no single dimension along which one model is simply better than another. Being good at retrieval and being good at clustering are different abilities, and a model can have one without the other. The single MTEB number exists only by averaging across that difference.

A model can lead the mean and be ordinary at retrieval

The table's last column shows the imbalance: retrieval holds 15 datasets, summarization holds 1. Nothing weights the average by how much a task matters, so it is weighted by how many datasets a task happens to have, and retrieval counts fifteen times as heavily as summarization for that reason alone. The authors say so plainly in their own limitations section.

This is a property of the mean, not a hidden scandal, and it cuts in both directions. A model can climb near the top of the overall average while being no better than ordinary at retrieval, because retrieval is one slice among eight and strong scores elsewhere carry it. A model genuinely tuned for retrieval can also sit lower than it deserves, dragged down by the categories a retrieval user does not care about. One independent writeup makes the second case directly. A model tuned for retrieval and semantic similarity, "the two categories most correlated with production performance in RAG and search," can "underperform on clustering or classification, which brings down the average."³

Either way, the overall rank and the rank on the task a model is actually being chosen for are two different measurements, and the leaderboard sorts on the first. Being ranked above most other models is not the same as being the best model for one particular job. The maintainers now build the leaderboard to be filtered down to the tasks that matter to a given use, and note that a single pre-defined board "might only contain 50% of the tasks that you care about."⁴ The overall number was never meant to answer the retrieval question on its own. It is read as though it were.

The authors changed their minds about contamination

The imbalance was on the record from the start. The second weakness was not, and its history is the more instructive one, because the same people reversed their position on it. Contamination is the worry that a model scores well because it was trained on data too close to the test. In 2022 the MTEB authors considered it and set it aside.

By 2025 the same team, revising its own benchmark, treats contamination as a first-order problem. In an engineering paper on maintaining MTEB, the maintainers describe a subtler failure than a test answer leaking into training: models trained "using training splits from the same source as benchmark evaluation tasks."⁵ To measure it, a model reports which benchmark datasets it trained on, and MTEB computes a zero-shot score: the share of the benchmark the model did not train on. Their worked case is e5-mistral-7b-instruct, which scores 95% zero-shot on the legacy English benchmark, meaning it trained on only about 5% of that benchmark's datasets. It still scores well, which the authors read as real ability rather than a leak.⁵ The point of the exercise is the models on the other end, where a high overall score sits next to a low zero-shot score. Outside groups note that "many embedding models are partially trained on data that overlaps with MTEB datasets," which "contaminates the leaderboard."⁶

The rebuild answers both weaknesses without throwing the old number away. For imbalance, the multilingual expansion MMTEB keeps the plain per-category scores and adds two things on top. The first is a ranking by Borda count, a method from voting theory the maintainers argue is sturdier than a raw mean. The second is an average weighted by task category.⁷ The flat average still exists; it is now one view among several rather than the only one. For contamination, MMTEB is built as a zero-shot benchmark that excludes the datasets models most often train on, naming MS MARCO and Natural Questions.⁷ And a newer board, RTEB, keeps some datasets private and scored only by the maintainers, so that "a model with a significant performance drop between the open and the private datasets would suggest overfitting."⁸

None of this makes the leaderboard a trick. The MTEB average is a real, usable summary of how a model does across a wide spread of tasks, and it was built for a reason. Both weaknesses are limits on how to read the number, not evidence that the number is fake: one task is a slice of the average, and a very high score can carry a little training on the test. The maintainers took their own measure seriously enough to keep repairing it, which is a reason to trust the repaired benchmark rather than to throw the whole thing out.⁷