Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation

Base Model Evaluation - LLM Benchmarking - Soft-Prompt Tuning

Blog post for arXiv:2606.12117. The figures on this page are interactive reconstructions of the paper's results, built for intuition. See the paper for exact numbers.

TL;DR

Are your LLM evaluation benchmarks measuring true model knowledge or just formatting skills? Often, current evaluations conflate model knowledge and format-following abilities: a model may possess the correct answer, but if it doesn’t say it in the expected format, it gets punished. When evaluating LLMs (particularly base models) it is crucial to isolate these variables and give them a fair chance to show what they actually know. The solution? Brief soft-prompt tuning. Short soft-prompt tuning removes this "formatting tax", separates knowledge from style, and gives you a cleaner signal about a model's true capabilities. Without this quick method, one would be forced to conduct full post-training to achieve the same behaviour which is a massively expensive process that eats up weeks of cluster time.

The LLM Evaluation Problem and Soft-Prompt Tuning as a Solution

Benchmarks serve as the standard metric for evaluating Large Language Models (LLMs) on diverse tasks. Yet, these scores frequently misrepresent actual model knowledge by conflating what it actually knows with its ability to adhere to strict formatting requirements. For example, a model may possess the factual knowledge that Berlin is Germany's capital, yet receive a score of zero solely because it outputs It is Berlin rather than the exact required string, #### Berlin.

This is especially significant for base models — models after pre-training but before instruction tuning or reinforcement learning — which often contain the relevant knowledge, but are worse at following instructions and producing benchmark-specific answer formats. Consequently, this leads to situations where the model possesses the requisite knowledge, yet the evaluation metric fails to capture it.

We propose to use soft-prompt tuning (Lester et al., 2021) as an efficient method for fairly evaluating LLMs across various benchmarks. Soft-prompts are learnable embeddings prepended to the model's input, essentially serving as a substitute for hand-written instructions. During tuning, we freeze the model and train only 10 small vectors (about 0.0006% of a 7B model's parameters) for ~80 steps (~1.5 minutes for a 7B model) on the benchmark's training split. We then apply these embeddings, learned from the training data, directly to the test split. This teaches the model the format of the benchmark, without changing the model's knowledge. As a result, benchmark scores better reflect the model's internal knowledge.

Common alternatives like log-likelihood scoring or post-hoc answer extraction work fine for fixed-choice tasks, but fail with open-ended tasks. Free-form generations often include extra reasoning, ambiguous or multiple answers, making it hard for evaluation tools to reliably extract the model’s true response. This often leads to incorrect or unfair scoring. See the paper for a fuller discussion.

Problem & proposed approach: a practical example

Here is an example from the GSM8K dataset, which contains math questions. The models below actually can do the arithmetic. Watch what happens to the benchmark score as you switch between a base model, a post-trained model, and their soft-prompt-tuned versions.

2 dozen seashells is 2 x 12 = 24 seashells. Kyle found 24 x 2 = 48 seashells. Leigh had 48 / 3 = 16 seashells.correct · not captured
Question: Mimi picked up 2 dozen seashells on the beach. Kyle found twice as many shells as Mimi and put them in his pocket. Leigh grabbed one-third of the shells that Kyle found. How many seashells did Leigh have?
Answer: 8wrong answer
Explanation: 2 dozen seashells is 2 x 12 = 24 seashells. Kyle found 24 x 2 = 48 seashells. Leigh had 48 / 3 = 16 seashells.
Question: Mimi picked up 2 dozen seashells...

Mimi has 2 x 12 = <<2*12=24>>24 sea shells.
Kyle has 24 x 2 = <<24*2=48>>48 sea shells.
Leigh has 48 / 3 = <<48/3=16>>16 sea shells.
#### 16

Both base and post-trained models fail to show their real knowledge in benchmark scores simply due to formatting quirks like looping. By tuning just 10 learnable soft-prompt vectors on the frozen model for a mere 80 steps, we bypass this formatting bottleneck entirely, unlocking the hidden knowledge within both base and post-trained models.

What is soft-prompt tuning?

A normal prompt is text: you write instructions and hope the model follows them. Finding the wording that unlocks a model's best performance is non-trivial, task-specific, and famously fragile — change one word and the score can swing.

A soft-prompt skips the words entirely. Instead of natural-language tokens, we prepend a handful of learnable vectors directly in the model's embedding space, right in front of the real input. We freeze the entire model and train only those vectors, leading the model to discover its own optimal prompt for the task.

For our method, we deliberately limited ourselves to training only 10 soft-prompt vectors for only 80 training steps (~1.5 minutes for a 7B model) on the benchmark’s training set before evaluating on the test set. We imposed this strict setup because it provides just enough capacity for the model to adapt to the expected format, without allowing it to learn new task knowledge.

Soft-prompt tuning maximizes format following and does not show additional knowledge gains

To independently track knowledge gain and format adaptation, we define two key metrics:

• Knowledge accuracy: Whether the model provides the correct answer, regardless of formatting. On multiple-choice tasks, we rank the answer options by log-likelihood and pick the best one. On open-ended tasks, an LLM-as-a-judge (Qwen3-32B) checks whether the correct answer appears anywhere in the response, ignoring formatting.
• Format-following accuracy: Whether the model responds in the expected format, regardless of factual correctness. On closed-ended tasks, we check whether the generated text exactly matches one of the predefined answer options. On open-ended tasks, we use the same LLM-as-a-judge to check whether the formatting is correct.

We designed the judge prompts against human-validated examples; full prompts and examples are in the paper.

To benchmark our results, we compare soft-prompt tuning against traditional zero-shot and few-shot prompting. The zero-shot baseline evaluates the model's performance using the base prompt followed directly by the question, whereas the few-shot baseline inserts n (in our case, n=3) question-answer example pairs from the benchmark's training split to demonstrate its formatting style.

As shown in the figure below, knowledge accuracy remains flat throughout soft-prompt tuning, while format-following accuracy climbs to 90–100%. This suggests that our approach maximizes format adherence without injecting new knowledge.

Closed-ended tasks saturate fastest (~step 40) because their format is simpler: copy one of the listed options with its letter (A. Berlin). Open-ended tasks take a bit longer (~step 80) because their formatting requirements are more demanding (#### answer, short trivia phrases, LaTeX). Therefore, we choose step 80 as the sweet spot to maximize format-following without knowledge gain.

Soft-prompt-tuned base models strongly predict post-trained model rankings

During LLM development, researchers typically generate numerous candidate base models using different pre-training recipes (e.g., varying data mixtures and architectures). Evaluating these candidates to identify the optimal recipe, however, is non-trivial, as fully post-training every variant is prohibitively expensive. To address this, we propose soft-prompt tuning as a computationally efficient proxy to accurately predict final post-trained model rankings without the need for costly full post-training.

Once formatting noise is eliminated, the performance rankings of soft-prompt-tuned base models should closely match those of fully post-trained models. To evaluate this, we measure the agreement between the base model rankings and the post-trained rankings using the bubble-sort swap distance — the number of adjacent swaps required to transform one ranking into the other. A lower swap distance indicates stronger agreement.

Importantly, any alignments achieved via zero- and few-shot methods are largely artifacts of score compression. These models yield near-zero exact match accuracy on benchmarks, failing to provide meaningful performance metrics. As a result, the sorting algorithm defaults to alphabetical sorting rather than score-based ranking, meaning these rankings are highly noisy and do not represent genuine real-world performance. In contrast, this limitation does not apply to soft-prompt tuning. As the plot below demonstrates, soft-prompt tuning elevates the base model's performance scores to levels comparable to the post-trained models, thereby avoiding score compression and yielding meaningful, reliable alignments.

The plot below shows the score gap between base and post-trained models across zero-shot, few-shot, and soft-prompt tuning methods. Confirming our earlier findings, soft-prompt tuning yields the smallest gap, suggesting it is the method that most effectively elevates base model performance to align closely with post-trained models.

The takeaway

A good evaluation practice involves making sure we actually measure what we set out to measure—without letting other factors skew the results. Right now, benchmarks confuse a model's true knowledge with its ability to follow formatting rules, making their scores misleading. Soft-prompt tuning fixes this by separating what a model knows from how it speaks. It gives you a clearer signal to compare models before you invest in expensive post-training.

Read the full paper: arXiv:2606.12117