GPT-4 answers the same exam 23 points worse in Telugu than in English

The same questions, 85% in English and 62% in Telugu

OpenAI took MMLU, a 57-subject multiple-choice exam, translated its questions into dozens of languages with a commercial translation service, and ran GPT-4 on each. In English the model scored 85.5%. The same questions in Spanish cost it a point and a half; in Arabic, five and a half; in Bengali, twelve. In Telugu the score fell to 62.0%.¹ One model, one exam, the questions matched item for item, and 23 points of accuracy went missing between the first language and the last.

The questions were machine-translated, and OpenAI says as much. Some translations drop information the model needs; others keep English proper nouns that make an item easier. So part of the gap comes from the translation, and only the rest from how the model handles the language.¹ And the same chart reads as progress. GPT-4 in most of these languages beats the English scores of the models just before it.¹ The gap is real, and it is shrinking, both at once.

The cost moves too. Billing runs per token, and the same request in these languages is broken into far more tokens. So a user in Andhra Pradesh can pay about five times what an English speaker in the United States pays for equivalent use of the model.² A worse answer and a larger bill, for the same message.

Two parts of the system decide how a language comes out. One is the training corpus, the text the model learned from. The other is the tokenizer, which cuts every message into the small pieces the model actually reads. Take them in that order. The corpus sets how much the model ever saw of a language; the tokenizer sets how much of your context and your money each sentence spends.

The model saw far less of almost every other language

How lopsided is the text? The clearest firsthand figure is old. When OpenAI documented GPT-3 in 2020, it reported that the training data was 93% English by word count, with the remaining 7% spread across every other language.³ No maker of a current frontier model discloses the same breakdown, so this is the best direct number available, and it describes a model two generations back. The evaluations that follow lean on it as a proxy for the closed models, because there is nothing newer to use.

Start one step upstream, at the raw web these corpora are filtered from. Common Crawl, the large public web scrape most of them draw on, records the language of every page it collects. In its mid-2026 crawl, 40.6% of pages are primarily English, and the next language, Russian, is 6.8%.⁴ English leads the open web by a wide margin, but 41% is a long way from 93%. That jump is added by the model's builders. English-favoring quality filters keep more English pages, and the English books, web text, and Wikipedia mixed in on top push English above its share of the raw web.³ So the model's diet is even more English than the web it was drawn from.

Less text in a language means a weaker model in it. The size of that training share also tracks the size of the gap, in a way the tokenizer alone cannot account for. In the MEGA evaluation, GPT-3.5 scored 72.1% on a sentence-matching task in French and 67.0% on the same task in Japanese. Its tokenizer splits the two languages at almost the same rate, so that is not what moved. What differs is the data: GPT-3's disclosed counts hold about 3.5 billion French tokens against 214 million Japanese.⁵

The tokenizer bills every other language in more tokens

The tokenizer is a fixed table of text pieces, learned once, mostly from English-heavy text. A sequence that turns up often in that text earns its own piece; a rare one is spelled out from smaller fragments. Because the table was fit mostly on English, English words tend to land in one or two pieces, and text in a language the table barely saw is broken into many.

You can measure this yourself. Take Article 1 of the Universal Declaration of Human Rights, which has an official translation in hundreds of languages, and run each through cl100k_base, the tokenizer ChatGPT and GPT-4 use. In English the paragraph is 33 tokens. In Spanish it is 44, in Russian 74, in Arabic 88. In Hindi it is 182, in Telugu 281, and in Burmese 512, which is 15.5 times the English count for the same sentence.⁶ Read as characters per token, English packs 5.15 characters into each token and Amharic 0.34, so on the scripts the table was least fitted to, the tokenizer is working almost one byte at a time.⁶

Why does a script fragment so hard? Petrov and colleagues trace one word. In Shan, spoken in Myanmar, the word for "you" is a single written character built from a consonant and three marks, which Unicode stores as four separate codepoints. The English-trained table has no piece for any of them, so it emits nine tokens where English "you" is one.⁷ The same paragraph, the same meaning, several times the pieces to carry it.

Two things follow, and both land on the same speakers. First, the model has a fixed context window measured in tokens. A Burmese user therefore fits a fraction as much text into one request as an English user does, and long inputs overflow sooner.² Second, the commercial APIs charge per token, so the Telugu or Amharic user pays for the extra pieces. Ahia and colleagues found that prompting plus generation cost up to four times more in those languages than in English for the same task. They estimate the Andhra Pradesh user paying about five times an American's price for equivalent use.² Petrov's group put the general penalty at a cost of at least 2.5 times, latency about twice as long, and an order of magnitude less usable context for the worst-served languages.⁷

It is tempting to read this off the script, as if Telugu or Burmese characters were simply expensive to write down. They are not. The premium is a property of what the tokenizer was trained on, and it moves when you retrain. MuRIL, a tokenizer built for Indian languages, encodes Telugu at 1.21 times English, and most Indic languages within about 1.06 to 1.26 times, against the fivefold and more those same languages pay under the English-centric table.⁷ Same script, different training, most of the tax gone.

Nobody can yet split the blame between the two causes

Two parts of this are settled. That most languages are underrepresented in training text is not in dispute, and neither is the token tax. A 2025 study ran the same tokenizer across more than 200 languages. It still finds non-Latin and morphologically complex languages paying three to five times the English tokenization cost, so the amplifier is current and not an artifact of one aging tokenizer.⁸ The pattern is well enough known that developer tutorials now warn engineers a sentence that is four or five tokens in English can run to fifteen or twenty in Hindi, Thai, or Burmese.⁹

What is not settled is how much of the gap each cause owns. The MEGA authors measured both against performance and declined to rank them. Tokenizer fragmentation correlates with worse scores, and training-data size correlates with better ones. The two also correlate with each other, and a correlation across a handful of tasks is not proof that either one causes the gap.⁵ The French-and-Japanese case is why data goes first: there the data differed and the tokenization did not, and the data won. But no source hands over a clean percentage, and Ahia's group cautions that even the token disparity is not always downstream of the data share, and may depend in part on features of the script itself.² Anyone who tells you it is 70% data and 30% tokenizer is making the numbers up.

And the gap is closing without being closed. GPT-4 bridges much of the distance older models left, which is the progress reading of that first exam. Yet the within-model gap remains. On the lowest-resource non-Latin languages a small model fine-tuned for the language still beats GPT-4, and translating the input into English first is often the best move available.⁵ Larger multilingual models and better-fitted tokenizers have narrowed the gap. Neither has closed it.