Parsy: Distilling an Agentic Parsing Harness into a Fine-Tuned Model

TL;DR

Our forward-deployed AI solutions hit a performance ceiling in customer projects because of poor data ingestion quality. Since no existing parsing solution combined the quality and the sovereign deployment our customers needed, we built an agentic parsing harness that checks its own output against the original page. The harness scored well, placing it at the top of public parsing benchmarks, but running several models and passes per page was too slow for high-volume work. So we used it to label training data, cleaned those labels with automatic checks and human spot verifications, and trained a specialized model on them. That model, Parsy, now scores 87.9 on ParseBench against 86.0 for the harness that taught it, parses hundreds of thousands of pages per day on a single H100, and costs one to two orders of magnitude less per page than the harness. It now powers the extraction work in our customer solutions at Aleph Alpha and drives source highlighting in our chat applications.

We did not set out to build a parser

Parsy originated from a pattern we kept seeing across our industrial and public customer projects. Before any search system, assistant, or extraction pipeline could answer a question, it had to rely on content parsed from datasheets, maintenance manuals, official forms, progress reports, test reports, and specifications. Those documents are dense with nested tables, footnotes, charts, and scanned pages. Whenever answer quality fell short, we traced the errors back through the pipeline, and most of them originated in the parsing step. A typical error case was complex tables. The parsers we used recovered every word and number in a table, but lost which value belonged to which unit and which row. In other cases they invented characters that were never on the page or attached footnotes to the wrong passages. The result was wrong or incomplete answers for the users working with our solutions.

A recurring requirement our customers have is that their documents stay inside their own environment, so any parsing solution has to be self-hosted, either on-premises or in a private cloud they control. No tool we tested offered both the quality and the sovereignty we needed. So we built our own.

Checking the parse against the page

Our idea was to automate what a careful human reviewer does. A diligent reviewer verifies extraction accuracy by placing the extracted output side by side with the original page. However, doing this manually is impractical at scale. We therefore built an automated harness that includes this exact comparison. The harness is built on three ideas.

First, every element keeps its position. The harness renders each page once, establishes the regions on it, their categories and their reading order, and extracts the content of each region. Every extracted element carries a bounding box, allowing the page to be reconstructed at its original coordinates.

Second, the original page is the reference. The harness screenshots that reconstruction, places it beside the original, and lets a vision model compare the two. Missing content leaves a gap, repeated text shows up twice, and a table that lost its structure stops looking like a table. A judge model then evaluates each comparison, passing on only pages with a high similarity score. Pages without a high similarity score are re-extracted and re-checked. Pages that remain uncertain are routed to an optional human reviewer, with the region and the judge’s note attached. This quality assurance requires no labeled data, as the original document serves as the ground truth.

Third, we enable more efficient human review. We have created our own front-end for manual quality checks and to offer transparency in a parsing pipeline. As every page carries a quality score and a note, human reviewers only open the pages that need their attention. The review interface shows the original page beside its reconstruction, and the same functionality is available as an API for teams that run it inside their own pipelines.

What real-world deployments taught us

We deployed the harness in customer applications across industry and the public sector through 2025 and 2026, and learned two things. First, the quality was good enough for our customers. Customer feedback said so, and so did public parsing benchmarks. On ParseBench, a public benchmark of 2,078 human-verified pages from real enterprise documents, the harness scored 86.0 in our internal evaluation. That score would have ranked first on the public leaderboard until the end of August 2026. Second, for customers with very large volumes the harness was too slow and too expensive. Every page goes through the judge at least once, and difficult pages go through the loop several times. For a customer processing millions of pages a month on their own hardware, that means more GPUs. We optimized the harness in many ways, but its underlying design stayed the same, using several models and up to three passes per page.

Further tuning yielded diminishing returns, and a larger model for the first pass would have driven up costs alongside quality. So we decided to train our own model with the single goal of matching the harness in one pass. Our existing harness serves as a natural source of high-quality training labels.

From harness to training data

We did not train on any customer data. Instead, we collected public documents in line with legal requirements that resemble what our customers work with, from technical datasheets and maintenance manuals to scanned forms and administrative documents. The harness labeled this corpus, with every page going through extraction, reconstruction and judgment, as it would in production. The target matches the harness output, which pairs every element with its category, bounding box, and reading order position. We trained a specialized vision-language model on this format.

The harness’s own judgment was only the first filter. A model trained on parser output inherits that parser’s systematic errors, and a judge that shares a blind spot will not catch them. So after each training round we checked the labels with tools that do not share the harness’s blind spots. An independent layout model proposes its own regions, and pages where its boxes and ours disagree get flagged. An ink-coverage check finds printed text that falls outside every labeled box, which means something was dropped. A second OCR model reads the same regions, and disagreements get flagged.

None of these checks is reliable on its own, and many flagged pages turn out to be fine. So people looked at samples from each detector to confirm what it actually catches before we acted on it. Then we fixed or removed that failure type across the whole corpus and trained again. Each round followed the same loop: training, evaluating on a held-out set, inspecting where the model lost the most points, fixing or adding data of that kind, and training again.

One example: in long lists and tables, some labels had merged several printed rows into a single box. The text was right, but the geometry was wrong, and the model learned to merge rows too. A check that compares the rows of ink on the page with the rows in the label found these pages, and once they were repaired the model stopped doing it.

The student passes the teacher

The first model version scored about two points below the harness on ParseBench. In the sixth round of training and label cleaning, the model passed it, and it has stayed above it since.

We attribute the model overtaking the harness to where each gets its quality from. The harness reaches its quality at run time, by combining several models and correcting their mistakes on every page. The model learns from the cleaned output of that process, so most of those mistakes never reach its training data. We also observed that passing Parsy’s output through the harness’s judge-and-refine loop lowers its ParseBench score. The judge now flags correct output more often than it catches real errors, so its corrections introduce more mistakes than they remove.

Results: one model, three benchmarks

We evaluated Parsy on three public benchmarks, each scored with its official code, and compared it with the highest-scoring published systems as reported in technical reports and on leaderboards. ParseBench is the enterprise document parsing benchmark mentioned above, olmOCR-bench is the standard OCR benchmark for English documents, and fr-bench-pdf2md is its French counterpart, focused on handwriting, forms and complex layouts.

ParseBench

Scored with the public ParseBench code, Parsy reaches 87.9 across all five categories, including visual grounding. That would place it first on the public leaderboard, above LlamaParse Agentic Plus at the reported 86.6. Parsy is not listed there because the leaderboard covers self-serve APIs, and Parsy is currently available through our customer engagements.

Almost every top system on ParseBench is a pipeline that runs several models and several passes per page, much like our original harness. Parsy is a single model and handles each page in one pass. Among vision-language models on the leaderboard, including general-purpose frontier models, the next-best score is 80.5, seven points below Parsy.

Reproducing ParseBench

While evaluating on ParseBench, we found questionable ground truth on 145 pages. We shared an interactive viewer with 316 clear cases with the benchmark maintainers, who are now preparing a new dataset version with corrected annotations and improved metrics.

Leaderboard numbers increasingly drive procurement decisions, so we also checked that they can be reproduced. On 27 and 28 October 2026 we re-ran Parsy, the three LlamaParse pipelines, Gemini 3 Flash, Nemotron-3.5-Parse, and LightOnOCR-3-4B with the public scoring code, using the LlamaParse EU API. The open-weight models matched their leaderboard entries to within about a point, but the LlamaParse pipelines initially did not. We shared our results with LlamaIndex, who traced the gap to differences between their internal evaluation and the public pipeline, and refreshed the LlamaParse scores on the leaderboard the same day. Cost Effective and Agentic now match our runs to within half a point. Agentic Plus remains about four points above our measurement, mostly on formatting and charts, and we are looking into the remaining difference together.

We are glad our findings could contribute to a more consistent and reproducible ParseBench. Alongside the refreshed scores and the upcoming dataset version, the ParseBench team is setting up scheduled runs and public result repositories, so that anyone can reproduce the leaderboard. We thank them for responding so quickly, and for creating and maintaining benchmarks and leaderboards for this field.

Throughput and cost

A single H100 parses hundreds of thousands of pages per day with Parsy. The exact number depends heavily on page density. For example, pages with long and dense tables take longer than simple text pages. Resolution and layout requirements also matter, because boxes and structure make up about a third of the output. Compared with the harness, the cost per page is one to two orders of magnitude lower.

What a parsing model enables beyond parsing

Knowing where every element sits on the page makes the model useful beyond extraction, for example for source validation in chat assistants. In most RAG assistants, a citation is just a document name and maybe a page number, so the user has to open the document and search for the passage themselves. As Parsy keeps the bounding box of every element, an answer can point to the exact region on the page it came from. The assistant shows the cited passage highlighted on the rendered page, so the user can check the answer against the source at a glance. The same boxes drive the review interface in Figure 3.

What comes next

Today, Parsy is available through our customer engagements, as a core capability that our forward-deployed engineers bring to customer projects and that our customers use in their own applications. If document parsing is a bottleneck in your organization, get in touch, and we are happy to show you what Parsy does with your own files.