Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

Checkpoint selection should account for the training that remains.

A large language model is built in stages, and after each stage we evaluate it to decide which versions to keep training. That saves a lot of compute, but it only pays off if today's ranking still holds at the end. Ours didn't: we ran three versions of one model through the same remaining stages, and some swapped places, so the early test would have kept the weaker one. Testing later helps, at the cost of keeping more models in later stages. What separated the versions was not a score, but how much random noise their weights tolerate before performance falls away.

Checkpoint selection should account for future training

Modern large language models (LLMs) are trained across several stages. These include pretraining, capability-focused mid-training, long-context adaptation, supervised fine-tuning (SFT), and reinforcement learning (RL). Ablating every model variant across the full stack through the late stages is very expensive. That’s why language-model checkpoints are usually compared by asking which one is better now, i.e., by comparing evaluations at the current stage.

An intermediate evaluation is therefore also a pruning rule. When we discard one checkpoint and continue another, we implicitly assume that their evaluation ranking remains invariant in future training stages.

We show that this assumption fails while training a 30B-total, 3B-active mixture-of-experts model, from which we derive three different pretraining checkpoints.

All three pretraining checkpoints come from the same constant-learning-rate run.

• Constant is the checkpoint after 7.5 trillion tokens, with the learning rate still at its peak.
• Cooldown forks after 6.7 trillion tokens and retrains on the same final 800 billion tokens that Constant saw, but decays the learning rate on a cosine schedule to 10% of its peak.
• Merge adds no training: it averages the final 20 checkpoints of the constant-learning-rate run1.

Checkpoint merging can mimic some of the effects of learning-rate decay2. The reason is simple: averaging checkpoints from a constant-rate trajectory effectively gives less weight to later parameter updates. However, Cooldown changes the optimization path during training, while Merge leaves the Constant trajectory untouched and combines checkpoints only afterward.

At the end of pretraining, Cooldown has a higher aggregate score than Constant. But after the same mid-training, long-context, and SFT pipeline, the ranking reverses. Constant finishes ahead of Cooldown, and Merge leads on both aggregate scores. Better pretraining scores alone therefore do not tell us which checkpoint will be the better starting point for subsequent training stages.

The obvious remedy is to evaluate as late as possible, but we show below that this works only partially. Mid-training aggregate scores stay weakly correlated with post-SFT aggregate scores, and long-context aggregate scores track them closely across our learning-rate sweep, yet still put Cooldown ahead of Constant. No intermediate aggregate score we measured separates these two pretraining checkpoints correctly, so we look for a property of the checkpoint itself rather than a later aggregate score.

One candidate is solution density: the fraction of Gaussian weight perturbations that retain task performance above a threshold. In our experiments, Constant and Merge have greater solution density than Cooldown. Related work finds that solution density increases with model scale and argues that denser neighborhoods of task-specific experts make useful specialists easier to reach through post-training3. Figure 3 visualizes solution density within a single model family.

These checkpoints are all the same size, so the differences in Figure 3 come from the trajectory rather than from scale. We therefore hypothesize that solution density also varies with how a checkpoint was reached, and that a wider performance-preserving neighborhood helps a checkpoint absorb the optimization that follows.

Stage transitions change what quality means

During all training stages, the next-token objective remains largely unchanged. Curriculum, context length, and loss mask determine what the model learns at each stage. The benchmark scores in this section all come from the Merge checkpoint under the matched $(1,1)$ recipe.

Our training recipe follows Olmo 34. The mid-training, long-context adaptation, and SFT datasets are derived from Dolma 3 Dolmino, Olmo 3 Longmino, and NVIDIA Nemotron Cascade, respectively.

Pretraining produces a text-completion model

Given a token sequence $x_1, \ldots, x_T$, causal language modeling minimizes

where $\mathcal{M}$ contains the token positions included in the loss. During pretraining, mid-training, and long-context adaptation, this happens for every token in a packed sequence.

Mid-training raises GSM8K accuracy from 0.455 to 0.782

Mid-training keeps the training objective but changes the curriculum to a mixture emphasizing math, code, science, reasoning, and multilingual data. The effect is visible in a GSM8K problem5:

The overall benchmark score increases from 0.455 to 0.782, showing that the improvement is not confined to this example. Neither checkpoint stops after answering, and both roll straight into the next problem. This is expected: every stage so far trains the model to continue a document, so nothing has ever taught it that an answer is a place to stop.

Long-context adaptation raises RULER NIAH recall from 0.047 to 0.917

Long-context adaptation retains the objective and capability-focused data, but adds long documents spanning roughly 8k to 64k tokens. Relevant information may now be tens of thousands of tokens away.

RULER is a benchmark family that measures whether a model can retrieve information from long contexts6. For instance, in the RULER needle-in-a-haystack (NIAH) task, one query places a special number (the “needle”) into a 117,351-character context of repetitive distractor text (the “haystack”):

The question near the end asks for that number.

The mid-training checkpoint misses the needle, whereas the long-context checkpoint retrieves it. However, it then carries on with distractor text, since nothing has yet trained it to stop.

SFT changes both the interface and the loss mask

SFT trains on an adapted conversational mixture defined by a chat template. The template introduces system, user, and assistant roles, and the loss applies to assistant targets only. It serializes a question as follows:

<|role|>system<|/role|>
You are a careful mathematical assistant.<|/turn|>
<|role|>user<|/role|>
A rectangle is 12 metres long and 7 metres wide. What is its area?<|/turn|>
<|role|>assistant<|/role|>
The area is 84 square metres.<|/turn|>

The loss set $\mathcal{M}$ contains the assistant response and the <|/turn|> marker. Everything before it is conditioning context and has no direct loss term.

Comparing checkpoints under a matched downstream recipe

During pretraining, Constant holds the peak learning rate it reaches after a 75B-token warmup. Cooldown forks at about 90% of that run. Merge is a post hoc weighted moving average of the final 20 Constant checkpoints, spanning 6.9T to 7.5T tokens at roughly 30B-token intervals, with newer checkpoints receiving linearly larger weights7.

Each pretraining checkpoint then undergoes the same three downstream stages: mid-training, long-context adaptation, and SFT. Every stage sets its peak learning rate as a factor of the preceding stage’s peak. We write a recipe as $(m,\ell)$ with SFT factor $s$, where $m$, $\ell$, and $s$ are the mid-training, long-context, and SFT factors, each taking a value in $\{1,1/3,1/9\}$. We sweep all nine $(m,\ell)$ combinations on Cooldown, holding $s=1/3$.

Recipe $(1,1)$ achieves the highest post-SFT aggregate score, so we reuse it for Constant and Merge. Compute constraints prevented tuning the schedule for each pretraining checkpoint separately, so the comparison uses a matched recipe selected on Cooldown without establishing that it is optimal for Constant or Merge.

Cooldown finishes with a lower post-SFT aggregate score than both Constant and Merge. Its largest deficit is on HumanEval+, which we examine below. It remains behind even when HumanEval+ is excluded, although the gap is smaller. The pretraining aggregate score therefore ranks Constant and Cooldown incorrectly for the downstream pipeline used here.

When checkpoint rankings become predictive

To understand when intermediate evaluations become useful, we use all eleven trajectories at SFT factor $1/3$: nine Cooldown $(m,\ell)$ recipes plus the fixed $(1,1)$ Constant and Merge recipes. At the pretraining, mid-training, and long-context boundaries, we evaluate completion-style benchmarks across knowledge, mathematics, and code, adding long-context retrieval from mid-training onward. For example, the shared suite includes MMLU and HellaSwag for general knowledge and reasoning, GSM8K for mathematics, and HumanEval for code; mid-training and long-context adaptation add tasks that require long-context understanding, such as RULER. We determine the pretraining aggregate score by averaging general, math, and code cluster means, while the mid-training and long-context aggregate scores also include the long-context cluster. After SFT, we directly average AIME, DAPO Math, Skywork Math, IFBench, HumanEval+, and GPQA. Since the benchmark suites differ by stage, we compare rankings and associations across stages, not absolute aggregate scores.

Across all eleven trajectories, Pearson correlation rises from $r=0.473$ ($p=0.142$) after mid-training to $r=0.884$ after long-context adaptation, while Spearman correlation rises from $\rho=0.482$ ($p=0.133$) to $\rho=0.964$, both with $p<0.001$ at the later stage. This improvement mainly reflects the learning-rate sweep: among the nine Cooldown trajectories, which start from the same pretraining checkpoint, the corresponding correlations rise from $r=0.448$ and $\rho=0.450$ to $r=0.935$ and $\rho=0.950$.

Even the aggregate scores after long-context adaptation misrank the pretraining checkpoints. Under the matched $(1,1)$ schedule, Cooldown’s long-context aggregate score is 0.637 versus 0.636 for Constant, yet it finishes 0.113 behind Constant after SFT.

Solution density separates the pretraining checkpoints

For checkpoint $\theta$, benchmark $b$, and Gaussian perturbation $\epsilon$, we define solution density3 at threshold $\tau$ as

It is the probability that a Gaussian perturbation retains at least a fraction $\tau$ of the checkpoint’s unperturbed score on benchmark $b$. We estimate it from 100 perturbations per pretraining checkpoint and benchmark on GSM8K and MBPP, adding independent Gaussian noise with absolute standard deviation $\sigma_\epsilon=0.005$ to every weight.

Cooldown shows lower solution density on both tasks. On GSM8K at $\tau=0.90$, solution density is 27% for Constant, 13% for Merge, and 0% for Cooldown. Figure 3 also shows that Merge is less prone to large performance losses than Cooldown. In line with Gan and Isola (2026)3, we hypothesize that this more robust neighborhood makes Merge easier to continue training.

Studying the neighborhood around a checkpoint is a common way to understand generalization and fine-tuning. For example, sharp minima have long been linked to poorer generalization8. In LLM training, reducing sharpness can improve downstream performance and reduce forgetting, while learning-rate decay can produce sharper checkpoints that perform worse after SFT910.

Solution density and sharpness both probe the neighborhood around a checkpoint’s weights: solution density tracks benchmark performance, while sharpness tracks changes in pretraining loss.

Retuning SFT does not close Cooldown’s performance gap

Cooldown’s largest final deficit is HumanEval+: 4.9%, compared with 64.6% for Constant and 66.5% for Merge. This score alone suggests a large loss of code capability, but the raw generations show a substantial stopping failure.

Cooldown shows similar failures on other benchmarks: AIME and GPQA responses, for example, almost always reach the 32,768-token cap, often with highly repetitive tails. On HumanEval+, the evaluator extracts the final fenced Python block. A model that writes a correct program and then repeats many failing blocks can therefore receive a low score despite producing a correct program earlier.

Because the failure appears after SFT, we tested whether retuning the SFT learning rate would repair it. For every one of the nine Cooldown mid/long checkpoints, we trained SFT factors $1$, $1/3$, and $1/9$, producing 27 complete checkpoints. Lower SFT rates improve HumanEval+ on average, but reduce reasoning and the overall aggregate score. Here, reasoning averages AIME, DAPO, Skywork, and GPQA.

In other words, no tested learning rate jointly resolves the failure. In particular, all 27 Cooldown checkpoints remain below the fixed-recipe Constant and Merge checkpoints. On the $(1,1)$ trajectory, increasing the SFT factor from $1/3$ to $1$ raises the aggregate score from 0.247 to 0.294, but leaves 100% of AIME and 99% of GPQA responses at the cap.

As a diagnostic, for the Cooldown $(1,1)$, SFT-factor-$1$ checkpoint, selecting the first generated code block instead of the final block raises HumanEval+ from 7.3% to 61.0% on the same 32k generations. This indicates latent code capability, but it does not repair the repeated output or failure to stop.

Implications for checkpoint selection

Three practical conclusions follow from the experiment.

1. Compare different pretraining checkpoints through the full training stack. Even after long-context adaptation, aggregate scores missed the Constant–Cooldown reversal after SFT. Solution density was a promising signal for this comparison, but still needs broader validation.
2. When sweeping learning rates, prune as late as your budget allows. In our Cooldown sweep, aggregate scores after long-context adaptation closely predicted the post-SFT ranking; mid-training aggregate scores did not.
3. Inspect failures behind aggregate scores. Cooldown’s low HumanEval+ score combined real assistant instability with substantial latent code capability. Both matter, but they imply different interventions.

Pretraining evaluations do not tell us which checkpoint will perform best after downstream training. Merge has the highest aggregate score before and after the pipeline and a larger solution density than Cooldown, making it a useful starting checkpoint in this experiment11. However, its final advantage over Constant is below 1%.

Limitations

This study covers one 30B-total, 3B-active mixture-of-experts model family, one seed per training setup, and an incomplete cross-product of pretraining checkpoints and downstream schedules. Constant and Merge use the best Cooldown mid/long schedule rather than independently tuned schedules, which favors Cooldown: its schedule is selected from nine configurations, or 27 including the SFT sweep, while Constant and Merge are each run once. Even the best tested Cooldown result (0.294) trails Constant (0.360) and Merge (0.363), so the observed reversal survives this sweep. Its magnitude and the Constant–Merge ordering may change under further tuning. Solution density is measured on only two benchmarks and does not establish causation. Cooldown’s stopping failure also magnifies the final gap in aggregate score. We audit that failure only on HumanEval+, so on the other benchmarks we cannot say how much of Cooldown’s deficit is weaker reasoning and how much is formatting or instruction following. The distinction matters, because the two call for different fixes. We do not include RL, which is one such fix: it could repair Cooldown’s repetition and reverse the rankings again.

Within those limits, the ranking reversal is controlled: the pretraining checkpoints receive the same downstream recipe, and the checkpoint that scored better before continuation produces the worse model after the downstream stages. Checkpoint quality therefore depends on the training that remains.

Further reading

Full experimental details are available in our paper “Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack”. If you find this work useful, please cite it:

@misc{maskey2026goodpretraining,
  title         = {Good Pretraining, Bad {SFT}: Checkpoint Quality Across the Training Stack},
  author        = {Maskey, Sohir and Scholl, Philipp and Knupp, Jonas and Neitemeier, Pit and Wirges, Sascha},
  year          = {2026},
  eprint        = {2609.08966},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.08966}
}