Translated Words, Untranslated Meaning: German Benchmarks Need More Than Translation

TL;DR: What does a translated benchmark actually measure? Often, something other than the original. We reviewed thirty German translations of popular LLM benchmarks and found three reasons why: translation errors, grammar mismatches, and questions still rooted in American culture.

Let’s say you want to test your large language model (LLM) with this riddle:

What can you call your sweetheart and also spread on toast?

In English, the answer is “honey”. In German, the riddle falls apart: the literal translation, “Honig”, is not a term of endearment. To test the same puzzle-solving skill in German, you would need a different riddle.

What an eval measures

If you have used a chat assistant, you have interacted with an LLM. To find out how good such a model is, we evaluate it. An LLM evaluation, also called an eval or a benchmark, is a collection of thousands of questions with known answers. The model answers them, and the percentage it gets right is its score. Common evals test topics like school science (e.g., ARC), everyday common sense (e.g., HellaSwag), or general knowledge (e.g., MMLU). Nearly every model release reports scores on evals like these.

What does such an eval score tell us? Broadly speaking, it reflects two things: what a model knows and what it can do. An eval usually targets one specific capability, but it is difficult to isolate capabilities cleanly. Take this real problem from GSM8K, a math eval:

Claire makes a 3 egg omelet every morning for breakfast. How many dozens of eggs will she eat in 4 weeks?

Solving it requires more than arithmetic. The model must also know that a week has seven days and a dozen is twelve. So even a math eval measures both what a model can do and what it knows. On top of that, the answer must come in the expected form1. No eval measures a single capability in isolation. Every score reflects a mix of capabilities, and this mix is what the eval measures.

In the end, we rarely care about individual questions, such as whether a model can count Claire’s eggs. What matters is what the score stands for: a mix of capabilities (here, math and world knowledge) that tells us how useful the model can be as an assistant. To interpret a score, we therefore need to know that mix. Translating the eval into German can change this mix, and with it what the score means.

Why German models need German evals

At Aleph Alpha, we build models for German-speaking users: models that work in German and in the context of German culture, law, and society. An earlier post argued that such models need German training data, and that translated English data is not enough. The same holds for evals. Yet nearly all standard evals are in English, and the widely used German versions are machine translations. Do these translations still measure what we want them to?

In our review of thirty German-translated evals we found many problems. Here we highlight three: (funny) translation mistakes; grammar that gives the answer away; and questions that stay rooted in the source culture, just asked in German.

When the German itself becomes the puzzle

The most obvious problem with translated benchmarks is also the oldest one: incorrect translations. Traditional machine translation systems are error-prone2. Mistranslations are also the simplest way a translation can change what an eval measures. We group them into three levels of severity, based on how much the original meaning survives3.

1. Wrong wording, same question. A single word or term is mistranslated, but the question stays intact. To answer, the model has to see past the bad translation, often by guessing the English word behind it. The score now partly measures how well the model can reconstruct the English original.

2. Clean wording, different question. Every word is correct German, but together the words ask something else. Nothing signals that the question has changed, so the model is scored against the answer to a question it was never asked.

3. Broken sentence, no meaning. In the extreme case, the translation is no longer German at all: the word order is wrong and the grammar is broken. Such an item no longer measures what it was meant to. At best, it tests whether the model can reconstruct the original question from broken German, a capability we never intended to measure. At worst, the question cannot be recovered at all, and the model can only guess. That adds noise: random variations in the eval score that say nothing about the model’s capabilities.

When grammar alone answers the question

The next issues are subtler and affect multiple-choice and fill-in-the-blank evals, where the model picks the option that best fits the question. English words barely change form: the articles “the” and “a”, for example, fit every noun4. In German, every noun has a gender, and articles and adjectives change form to match it. So in German, the beginning of a sentence restricts how it can continue. For instance, in the English fill-in-the-blank I ate the ___ (hamburger / melon), both options fit. In the German translation Ich aß den ___ (Hamburger / Melone), only “Hamburger” is grammatically correct, as “Melone” would require the feminine article “die”. Translating a question and its options separately therefore breaks their agreement, either for all options or, worse, for only some. This leads to the following problems.

1. Question and options drift apart. When question and answer options are translated separately and then recombined, they may no longer fit. In the ARC and HellaSwag examples below, the grammar breaks: no option continues the German question grammatically. In the two MMLU examples (Global-MMLU and Eurolingua), the content breaks: only part of the item was translated. Either way, the model first has to guess how the item was meant before it can answer it.

2. Label leakage. In some cases, the grammar even gives away the answer. In the ARC example below, the question ends in mit einem, only allowing the masculine and neuter nouns Schalter and Kabel (including the correct answer). Grammar has halved the choices before any physics is needed. The task gets easier, and the score overstates the model’s physics knowledge.

The reverse can also happen: assume a fill-in-the-blank question with two possible completions to choose from. One is grammatically correct but logically wrong, and the other one vice versa. This creates an ill-posed test, where the model is forced to choose between logic and grammar. Scoring such questions rewards a trade-off that no real user would make; answers are expected to be grammatical and logically coherent at the same time.

A simple fix is to move the article from the question into the options, as in the Wino-X example below. Each option then brings its own article, so all options stay grammatical. (English can keep “the” in the question because it fits every noun.)

German words, foreign context

The number 24.678 means different things in different places. In the US, it is a decimal number close to twenty-five. In Germany, where a period separates thousands, it is close to twenty-five thousand. Evals are full of such cultural context: names, units, places, and facts that belong to a particular region. A translated question keeps the context of its source, so it also measures familiarity with that region. This adds another ingredient to the mix.

Questions about baseball and US law are fair game for a German model, but they test knowledge of American life. Swapping baseball for football and hot dogs for Schnitzel would instead measure knowledge of life in Germany5. Both are valid tests. What matters is that this choice is made intentionally, not as a side effect of translation.

For our models to serve German users well, we want evals set in German life: if we only ever test a model on miles, gallons, baseball, and US law, we do not know whether it can handle kilometers, liters, football, and German law. But adapting an eval takes more than swapping words, as two examples show.

1. On the difficulty of translating units. Many evals, particularly in math and science, feature units. Some can simply be swapped: in the GSM8K duck-egg problem below, two dollars per egg could just as well be two euros. Converting the price at today’s exchange rate to, say, “1,72 Euro” would be contrived, not more German. Other units can only be swapped if the numbers change too. TruthfulQA asks for the smallest country that is at least one square mile in area. One square mile is 2.59 square kilometers, an oddly specific threshold hardly anyone would ask about. But rounding it down to two square kilometers would change the correct answer from Nauru to Monaco. Translating units takes Fingerspitzengefühl (a feel for the situation): knowing when to just swap the unit, when to convert the numbers too, and when to leave the unit alone.

2. Culture, law, and history do not translate. Some questions are explicitly tied to a country’s institutions. MMLU lists US foreign policy among its 57 subjects, and its legal questions are primarily about American law. A translation still asks about the American legal system; testing German law requires writing new questions. The same holds for culture: a translated question about a 1940 baseball championship remains a question about American sports history, and even a basic ecology question in ARC is set among alligators in Florida’s wetlands.

Smarter translation closes some gaps, not all

Recent agentic translation pipelines, such as the one behind ellamind’s multilingual datasets, have raised the quality of German evals considerably. Unlike previous systems, an LLM agent sees the full question, is aware of the evaluation context, and works in several steps: it translates, reviews its own translation, and adapts tone and wording. This agentic approach resolves many of the issues described above (wrong words and grammar drifts), as the examples show.

Translation note: Used 'Welches der folgenden Beispiele...' to accommodate the plural choice 'Bäume' (Trees) alongside singular choices.
Adaptation note: Translated 'Oil' to 'Erdöl' (Petroleum) to be scientifically precise in the context of natural resources.

Translation note: Translated 'cancer' idiomatically as 'Krebsgeschwür' (cancerous ulcer/tumor), which is the standard German translation for this metaphor.

In the ARC example, “oil” becomes “Erdöl” rather than “Öl” because the question is about natural resources. In the SimpleQA example, the “cancer” metaphor becomes the idiomatic “Krebsgeschwür”. The grammar drift disappears too: the ARC question was rephrased to “Welches der folgenden Beispiele” so that the plural “Bäume” fits the question alongside the singular options. In each case, the translation accounts for the context.

The pipeline also makes judgment calls about how to translate units, notation, and names mentioned in the previous section. 12-hour times become 24-hour times and names are localized (“Rani” becomes “Ronja”). Where the arithmetic depends on the unit, the unit stays: the GSM8K problem below keeps its miles per hour. Numbers in the answer options keep their original format, so that scores stay comparable with the original.

Adaptation note: Changed the name 'Rani' to 'Ronja'. Corrected the English typo 'faster car' to 'fastest car' ('schnellste Auto') for logical coherence. Kept miles and MPH. Changed 'times faster' to 'mal so schnell wie' to correctly match the mathematical operations in the solution.

Adaptation note: Adapted 12-hour time format (05:01 pm, 07:30 am, 5:00 pm) to 24-hour time format (17:01 Uhr, 07:30 Uhr, 17:00 Uhr) which is standard in German business contexts.

The limits appear in everyday scenarios. Here, the pipeline swaps foreign concepts for German equivalents. A physics question about American football becomes one about Fußball (see the first example below). That is the kind of adaptation the previous section asked for. The problem is that nothing controls how far it goes. In about a third of the evals translated this way, the pipeline frequently replaced whole scenarios6. At that point, the LLM is no longer only translating the benchmark but rewriting it7. This is not a problem per se, but it means the translation system decided for us whether the eval tests life in Germany or in the source culture.

Adaptation note: Adapted 'football kicked off a tee' (US sports context) to 'soccer ball kicked from penalty spot' (Elfmeterpunkt) to ensure the concept of 'object at rest' is culturally clear to German speakers without ambiguity. Kept 'Baseball' in choice B as it is a distinct scenario often cited in physics examples despite being less common in Germany.

Synthesis note: The scenario was shifted from 'Baking cookies' (indoor/domestic) to 'Making stick bread' (outdoor/recreation) to fit the requested Sports/Recreation category while maintaining the 'Moderate' creativity level (related activity of preparing food, but in a hobby context). The core logic of checking/finishing a baking process remains. Culturally specific elements like 'Haselnussstöcke' and 'Stockbrot' were used.

The notes also show that the pipeline follows fixed instructions, for instance to use stereotypical German names such as “Leon”, “Finn”, and “Ronja”. Applied across thousands of questions, such rules give the eval a far narrower range of names than people in Germany actually have.

More than a translation

A good German eval is more than just a good translation of an English one. Having examined thirty translated German evals, we found that translation often changes what an eval measures. Traditional machine translation introduces mistranslated words, broken sentences, and questions that no longer fit their answers. A wrongly translated eval then requires capabilities we never intended to measure, such as guessing the English original. Agentic translation pipelines largely fix these errors.

The biggest challenge remains: the cultural context that every eval carries implicitly. Should a question be about baseball or Fußball, the Supreme Court or the Bundesgerichtshof? The answer decides what the eval measures. These are value-laden decisions, and they should be ours, not the translator’s.

At Aleph Alpha, we build models for what our German customers actually need, from units and terminology to cultural and economic context. Our evals must reflect these needs. That means deciding ourselves what our evals cover and setting the focus on areas we care about, for which our German cultural knowledge eval Kulturbank is one example. These considerations are part of a wider suite for determining how to build high-quality evals, such as why differently tokenized spaces matter (Mind the Tokenizer) and how tweaking bits-per-byte scoring improves evaluations (PriorBPB). This way, we hope to evaluate German LLMs more faithfully and, in turn, build better models.

Disclaimer

The dataset examples in this post are short excerpts from publicly available third-party datasets, reproduced for illustration purposes only and subject to their original licenses. They were selected to illustrate specific translation issues and do not represent an assessment of the overall quality of these datasets or the work of their authors.

Acknowledgements

This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM).

The contents of this publication are the sole responsibility of the OpenEuroLLM consortium and do not necessarily reflect the opinion of the European Union.

Thanks to Teryn Jones and Emily Savine for helpful feedback.