Mind the Tokenizer: Why a Single Space Decides What Multiple-Choice Evaluations Measure

TL;DR: Evaluating Large Language Models with multiple-choice questions is highly susceptible to minor formatting details. We analyzed a peculiar case, where placement of a single space changes the probability the model assigns to the correct answer by several orders of magnitude, but its effect on the accuracy is, in most cases, small. We trace the root of this phenomenon to the difference in tokenization between training and inference.

A common convention in multiple-choice evaluations is to end the prompt with Answer: and read off the model’s next-token probabilities for the option labels A, B, C, D. As in ordinary written text, a space follows the colon, so the string reads Answer: B and not Answer:B. That single space has to belong to one side or the other, and a recent paper1 asks what may seem like an irrelevant detail:

Sanz-Guerrero et al. find empirically that Space-in-Answer scores consistently higher across a range of models, and recommend it as the default. See below for an illustration of the two ways of splitting the prompt and the choices.

Our new paper traces this effect to the tokenizer. With Space-in-Prompt, the correct answer is several orders of magnitude less likely than with Space-in-Answer. However, the effect on accuracy itself is much smaller, because the drop affects all candidates similarly. In six of nine model-benchmark pairs, accuracy changes by less than half a point, though Qwen3.5 loses almost ten points on PIQA. We note an additional effect: scored as bare letters, models favor some labels over others, such as B for Qwen3.5 and D for Gemma 4. This affects performance on a few benchmarks, mainly when the model is less confident. In this post, we explain how one source produces both effects, and why Space-in-Answer is still the right default.

A Primer on Pre-Training Evaluations

During pre-training, a model learns to continue text, not to respond to questions. Given “What is the capital of France?”, it may well continue with another question. Measuring the performance of a pre-trained model needs more than feeding it a question and checking the answer.

There are two common ways around this. Multiple-choice (MC) shows a fixed set of labelled options and asks which label, ␣A through ␣D, the model finds most likely. Cloze instead measures the probability of the answer strings themselves. In both cases, the probability of the answer options is measured instead of generating new text. This is likelihood-based scoring, and it is a useful tool for measuring what a model knows before it can produce coherent responses. It also remains useful later, for tasks where a model can pick the right answer but not yet generate it.

Question: What is the capital of France?

Answer:

Question: What is the capital of France?

A. Berlin
B. Paris
C. Madrid
D. Rome

Answer:

Question: What is the capital of France?

Answer:

The option with the highest probability is treated as the model's answer. A question counts as correct when that option is the right one, and the fraction we get right is the accuracy on that benchmark.

The Prompt–Answer Boundary Problem

During pre-training, the model sees data like

What is the capital of France?
A. Berlin
B. Paris
Answer: B

This string is tokenized, and the model is trained to predict the next token autoregressively. We call the tokenization that emerges here the canonical tokenization. Fig 3 shows it for six modern tokenizers: all of them split the text as Answer, : and ␣B. So the model is trained to predict ␣B given the question. On a one-million-document sample of DCLM-Baseline2, this canonical tokenization appears roughly 2,300 times per million tokens, whereas the alternative — a standalone ␣ followed by a bare B — appears about 0.03 to 0.10 times per million, a difference of four to five orders of magnitude.

At evaluation time, we have two options for splitting the string into prompt and answer.

• With Space-in-Answer, the prompt ends with Answer:, which tokenizes as Answer, :. The next-token prediction over the candidates (␣A, ␣B, …) then matches what the model saw in training.
• With Space-in-Prompt, the tokenized prompt ends with a standalone space. The model is expected to predict a bare B, which results in an overall tokenization Answer, :, ␣, B that the model has practically never seen during pre-training.

The split would not matter if tokenizing the prompt and the answer separately and then concatenating were the same as tokenizing the combined string, that is, if the following relation held:

Here $T$ is the tokenizer, $p$ the prompt and $c$ the candidate option; $+$ denotes string concatenation and $\circ$ token-sequence concatenation. We call this condition tokenizer composability, and Fig 3 shows that all the tokenizers we examined break it at the prompt–answer boundary.

Impact of Violating Tokenizer Composability

To study the impact of violating tokenizer composability, we compute the log-probabilities of the correct answer under the two ways of splitting:

• Space-in-Answer: log P( T("␣B") | T(p) )
• Space-in-Prompt: log P( T("B") | T(p + "␣") )

We compute these values for three datasets (MMLU3, ARC4 and PIQA5) and three base models (Olmo-3-7B, Qwen3.5-35B-A3B and Gemma-4-26B-A4B), and plot their histograms.

On Log-Probabilities of Answers

The difference between the splits is apparent and consistent across all models and benchmarks. With canonical tokenization, the histograms concentrate near zero, indicating that the model assigns most of the probability to the correct label. Without it, as in Space-in-Prompt, the log-probabilities are lower and spread wider.

For practical evaluations, this means that any quantity built directly on these likelihoods — for example the bits-per-byte6 of the correct answer — is also influenced by the choice of split. Whether this distortion affects all candidates equally, and so leaves their ranking and with it the accuracy unchanged, is what we test next.

On the Accuracy of Prediction

Given that Space-in-Prompt lowers the likelihood of the correct answer, one might expect benchmark accuracies to suffer as well. The table below shows the accuracy of likelihood-based evaluations for both splits.

We find that the drop in the log-probability is not only for the correct answer, but also for every option by roughly the same amount. Due to the uniformity of the drop, for six out of the nine pairs, the accuracy changes very little.

However, the drop is not exactly the same for every option. Some labels lose more log-probability than others. For Qwen3.5 on PIQA, A loses more than B, and Space-in-Prompt picks B for two thirds of the questions. If we remove this per-label difference, which we estimate from the Space-in-Answer scores, Qwen3.5 recovers almost nine of the ten points it loses on PIQA. Gemma 4 similarly prefers D on ARC and MMLU. Such label preferences change the accuracy mostly when the model is less confident. On ARC, where the models are usually confident, the accuracy barely changes. The details of these experiments are in the paper.

With this finding, we refine the main recommendation of Sanz-Guerrero et al.1 Space-in-Answer is not preferable because it results in higher scores — in the table above it usually does, mostly by small margins. It is preferable because it evaluates the model on the canonical tokenization it was trained on.

Finding the Missing Probability Mass

If Space-in-Prompt lowers the probability of all the options, what does the model assign probability to? Using samples from the ARC benchmark, we prompted Olmo-3-7B with Space-in-Prompt and generated ten tokens greedily. A visual inspection of a handful of samples shows that the model still produces a space — it generates ␣B even though the ␣ is already part of the prompt.

To check whether these completions are still semantically correct, we used another LLM (Qwen 3.8-27B) as a judge. The judge finds 90.25% of the completions correct, about 1.4pp above Olmo 3’s Space-in-Prompt accuracy on ARC. The missing probability has not moved to a semantically different answer. The model prefers to produce the canonical tokenization of the answer even after a prompt that already ends in a space.

An evaluation using Space-in-Prompt consequently conflates two very different questions: whether the model knows the answer, and whether it tolerates a tokenization it rarely saw in training.

The paper also studies the number of few-shot examples and the formatting of the option labels, additional space-assignment experiments, and additional discussion on how this ties to a large existing body of work.

Discussion

The phenomenon of the space sensitivity of prompts has been noted in several works before, under the names partial-token or prompt-boundary problem7,8. Suggested fixes include token healing for generation9 and BPE-dropout for more robust tokenizer training10. Our work ties this body of work to the findings of Sanz-Guerrero et al.1

How Existing Frameworks Handle the Boundary

The widely used lm-evaluation-harness11 moves trailing whitespace from the prompt to the candidate, tokenizes the combined $p + c$ string and takes the first $|T(p)|$ tokens as the prompt tokens. For a prompt ending in http with the candidate s://…, this turns the prompt into https, so the s is never scored12. In our own eval-framework13, we take the last $|T(c)|$ tokens instead, which implicitly moves the trailing whitespace from the prompt to the candidate. Both frameworks handle the boundary with heuristics, and neither checks whether the split corresponds to the canonical tokenization.

Recommendation

So how should we split the prompt and the options? Our recommendation is: whichever way results in the canonical tokenization. Which one that is can be determined before any inference, with access to nothing but the tokenizer, by testing whether

Further Reading

The full paper is available on OpenReview. If you find this work useful, please cite it:

@inproceedings{s2026mind,
  title     = {Mind the Tokenizer: Why a Single Space Decides
               What Multiple-Choice Evaluations Measure},
  author    = {Prabhu Teja S and Frank Schneider and Martin Reinhardt and
               Fabien C. Y. Benureau and Thomas F Burns},
  booktitle = {NeurIPS 2026 Trust-AI-Eval Workshop},
  year      = {2026},
  url       = {https://openreview.net/forum?id=RROuGABnBG}
}

Acknowledgements

This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM).

The contents of this publication are the sole responsibility of the OpenEuroLLM consortium and do not necessarily reflect the opinion of the European Union.

Thanks to Andreas Hartel, Helena Treeck, Yasser Jadidi, Svenja Fahlisch for helpful feedback.

Thanks to Alexander Wortmeier for bringing the blog to life.

References

1. Sanz-Guerrero, Bui, von der Wense (2025). Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs, arXiv:2509.15020. https://arxiv.org/abs/2509.15020
2. Li, Fang, Smyrnis (2024). DataComp-LM: In search of the next generation of training sets for language models, arXiv:2406.11794. https://arxiv.org/abs/2406.11794
3. Hendrycks, Burns, Basart, Zou, Mazeika, Song, Steinhardt (2021). Measuring Massive Multitask Language Understanding, ICLR. https://arxiv.org/abs/2009.03300
4. Clark, Cowhey, Etzioni (2018). Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge, arXiv:1803.05457. https://arxiv.org/abs/1803.05457
5. Bisk, Zellers, Le Bras (2020). PIQA: Reasoning about Physical Commonsense in Natural Language, AAAI 2020. https://arxiv.org/abs/1911.11641
6. Gao, Biderman, Black (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv:2101.00027. https://arxiv.org/abs/2101.00027
7. Phan, Havasi, Muckley, Ullrich (2024). Understanding and Mitigating Tokenization Bias in Language Models, arXiv:2406.16829. https://arxiv.org/abs/2406.16829
8. Athiwaratkun, Wang, Shang (2024). Token Alignment via Character Matching for Subword Completion, Findings of ACL 2024. https://aclanthology.org/2024.findings-acl.929/
9. Lundberg, Ribeiro (2023). The Art of Prompt Design: Prompt Boundaries and Token Healing, blog post. https://towardsdatascience.com/the-art-of-prompt-design
10. Provilkov, Emelianenko, Voita (2020). BPE-Dropout: Simple and Effective Subword Regularization, ACL 2020. https://aclanthology.org/2020.acl-main.170/
11. Gao, Tow, Abbasi (2024). A Framework for Few-shot Language Model Evaluation, EleutherAI lm-evaluation-harness. https://zenodo.org/records/12608602
12. Mistral AI (2024). Tokenization Boundaries, documentation. https://docs.mistral.ai/resources/cookbooks/concept-deep-dive-tokenization-boundaries
13. Aleph Alpha Research (2026). eval-framework, GitHub repository. https://github.com/Aleph-Alpha-Research/eval-framework