RL environments are an infra problem

On 3rd October, we released Kolibri, an in-house LLM post-trained with RL on 1.2 million tasks across 38 environments. When we started, little was public about running environments at training scale, so we built our own stack. Our design choices for the stack were heavily driven by the scaling issues we faced.

The scaling issues showed up in the order of capabilities we chased, starting with math and code, where an environment was a parser and a few tests.

One Python function can idle every GPU

Kolibri is trained asynchronously. At any moment, the inference workers have thousands of rollouts in flight, and the trainer learns from the ones that have finished. Both run on GPUs and everything on the environment side, (parsers, verifiers, sandboxes and the glue between them), runs on CPUs that cost a rounding error by comparison. That asymmetry gives the first rule.

GPUs that wait for the environment waste compute, and it takes only one blocking CPU call to makes them wait: a verifier stuck in a regular expression, a sandbox waiting for an image, a callback holding the event loop. If too many rollouts wait for the blocking environment operations, the GPUs get underutlized.

It first started happening in simple math and code environments. For example, a LaTeX parser or symbolic equality check runs on whatever the policy writes, which could loop forever. In our first design, these checks ran inside the same Python process that managed every rollout. A stuck check did not just lose the rollout but stalled the process which in turn stalled other rollouts in that process. Now, each check runs in an isolated process with a timeout, so a stuck check costs only its own rollout. That rollout scores zero, because a hang on the policy's output is the policy's fault.

A parser is the cheapest environment there is. Tool calling, assistant tasks and agentic coding each added significant infrastructure behind it: judge models, simulated users, search services, and sandboxes. The challenge was to keep these services up during training, when thousands of rollouts use them simultaneously.

Self-hosting

In the current state, the training job is self-contained with environment services, judges and sandboxes. In general, we observed this pattern of leveraging external services, running into their limits when scaling up, and ending up self-hosting them, usually inside the training job itself.

Services and Judges

To stay close to production, we first pointed environments at the production services themselves. That dragged in machinery that had nothing to do with the task: authentication and access control, real user traffic competing with rollouts, and releases that changed the service under a training run.

Similarly, for judges and user simulators, we initially went against an external inference service but the costs quickly added up, making it economically unviable. Moreover, the rollout traffic also overwhelmed the deployment. So, we started self hosting the judges and RAG machinery.

Sandboxes

We started with external sandbox providers so we could build environments first and worry about infrastructure later. That worked at a few hundred rollouts. Past that, the providers throttled sandbox creation and rollouts queued for code execution.

So we built our own container runtime for sandbox execution, which runs on the training job's compute nodes. Each run brings its own sandboxes, started by the controller and torn down with it. This helped us isolate the sandbox compute to each training job.

Sandboxes are usually discussed as an isolation problem. On our own nodes, they first became a storage problem. Software-engineering (SWE) datasets usually ship one container image per task. For context, the final Kolibri run trained on around 450,000 SWE tasks. Overall, this resulted in millions of sandboxes spun up and torn down, with an average of 25,000 sandboxes being active at any point of time.

Pulling images onto each node would have four failure modes: the registry becomes a dependency of the training run, pull latency sits on the critical path of every rollout, the images would be duplicated on each node's disk, and the full set of images is too big for any single node's disk.

So we never extract an image onto a node's disk. Each image is converted exactly once into a SIF file (Singularity Image Format, the single compressed read-only file that Apptainer runs) on shared scratch storage. On each node, that file is mounted read-only through squashfuse and shared by every sandbox that uses the image. And each sandbox gets its own writable layer through an OverlayFS layer backed by a size-capped tmpfs. For more details, refer to our tech report.

Because Apptainer assembles the filesystem from stacked layers, this gives us the flexibility to inject custom packages without having to rebuild the task images. For example, we train with several agent harnesses, and each makes its own assumptions about the image. One such harness needs tmux, which most task images lack, and installing it over the network in each rollout is wasteful and flaky. So, a static tmux tool layer is stacked on top whenever this harness runs.

A crashed sandbox looks like a wrong answer

In the GPU stalling section, we spoke about an environment failure scoring zero, because the hang came from the policy's output. That failure was easy to attribute.

For Sandboxes, it becomes more nuanced because a sandbox can fail due to broken code by policy mistakes or due to infrastructure issues, . From the outside, both can look the same. Early on, when our sandboxes were still flaky, errors were not attributed to their source. An infrastructure failure was scored like broken code, and the model was penalized for a correct solution.

A stall at least shows up on a dashboard. A failure scored as a wrong answer does not: the GPUs keep working, at full price, on a wrong signal.

Now, every rollout failure has an explicit outcome, decided by its failure mode. A policy failure, like the stuck check, scores zero like any other wrong answer. Transient infrastructure failures like a dead sandbox, unparseable judge verdict and broken search responses are dropped from training . Unclassified errors are also treated like infrastructure failures and dropped from training but they are logged for later analysis.

The model finds every hole

In general, sandbox escapes and reward hacks are just as much about sandbox security as they are about model capability. We encountered multiple such incidents during the development of Kolibri-1.

Reward hacking

When we switched to a harness with web fetch tools for software-engineering benchmarks, we saw a significant increase in performance. Later, we found that the model looked up upstream pull requests of the solutions and downloaded the patch file. Then, we introduced an egress proxy between every sandbox and the internet that decides which domains an agent may reach, and web fetching is gone from evaluation harnesses entirely.

During the model development, we found more reward hacking incidents. For our SWE tasks we had cut the local history back to the task's base commit, but the remote-tracking refs still pointed at the upstream branch, and the objects they referenced were still in the git folder. The policy only had to compare its checkout against the upstream branch to see the fix.

Other hacks include editing conftest.py so the failing tests were skipped, mocking the actual function, or printing the output the test compared against. Since the agent could change any grading material in the sandbox, we always apply the candidate patch in a fresh sandbox.

These issues are frequently encountered by the community. For example, Xiaomi open-sourced their RL environments and Vals AI found that in two-thirds of the coding tasks the answer was still in the task's Git history. As Zhuokai Zhao pointed out, deleting the branches had left the fix commit in the repository, and the files the fix touched had newer timestamps, which showed the model where to edit.

Sandbox escapes

The settings change incident from the start of this post began with model trying to set up a Python virtual environment, by running apt-get install python3-venv. The install failed because apt hands downloads to a separate unprivileged user, which was not allowed by the sandbox's user namespace. The model read the error correctly and traced it to user namespaces. As it was root inside the sandbox, it was able to disable user namespaces by writing to the kernel settings. The change actually propagated to the host kernel as our sandbox isolation was insufficient. Because sandboxes relied on the creation of unprivileged user namespaces, the affected host machine could not start sandboxes anymore.

Looking back, this was a pure infrastructure failure and the model itself was actually debugging the task to install dependencies. Sandboxes now run as unprivileged users: root inside the sandbox maps to an unprivileged range of host users, so package managers still work in the sandbox's own writable layer, but nothing can touch the host kernel.

Conclusion

We set out to train a model, thinking that environments and underlying data would consume majority of our time. However, we spent more time on the infrastructure underneath them, and that is where the lessons that stuck come from: keep the GPUs ahead of the environments, never swallow an error, and assume anything the model can reach will be reached. The tech report has the rest of the post-training story.