What the numbers mean
This is a small, inspectable benchmark. It does not establish a universal best PDF parser.
The scoring pipeline
What each fixture breaks
| Fixture | Table | Reading order | Heading | OCR routing |
|---|---|---|---|---|
| text-layer.pdf | · | · | ✓ | ✓ |
| tables.pdf | ✓ | · | ✓ | ✓ |
| ragged-table.pdf | ✓ | · | ✓ | ✓ |
| multi-column.pdf | · | ✓ | ✓ | ✓ |
| scanned.pdf | ✓ | · | ✓ | ✓ |
| interleaved-columns.pdf | ✓ | · | ✓ | ✓ |
| nse-market-pulse.pdf | · | · | · | ✓ |
6 single-page PDFs, each built to break one thing, plus one real report. Every parser runs three times on the synthetic fixtures and we report the median; a failure stays visible rather than being retried away. Source files and their SHA-256 hashes are published with the results, so you can check the file you download is the file we measured.
How each axis is scored
Cell alignment
Did each value land under the right column header?
- 100
- every cell under its expected header
- 0
- plain text, or values under the wrong header
A parser can keep every word and still score zero. That means it returned text, not a table — not that it lost content.
Reading order
Was each column read top to bottom, not zigzagged across the page?
- 100
- all of the left column, then all of the right
- 75.6
- the two columns zipped together line by line
Every pair of lines is checked, not just neighbours. Checking only neighbours would score that same zipped page 97.4%.
Heading structure
Did the title come out as a Markdown heading at the right level?
- 100
- a heading at the expected level
- 50
- a heading at the wrong level
- 0
- plain text, or missing
A text-only parser scores zero here by design. Setext underlines and HTML heading tags are not counted.
| Scoring method | Score on interleaved output |
|---|---|
| Adjacent pairs only | 97.4% |
| All pairs (used here) | 75.6% |
The two fixtures that score nothing
is the tables fixture flattened into an image with no text underneath, like a photocopy. Reading it needs optical character recognition — software that reads letters out of pixels. None of the six free parsers recover the table, and most return nothing.
nse-market-pulse.pdf is a real document: the first 100 of 330 pages of Market Pulse, September 2026 (Volume 8, Issue 9), from NSE India’s Economic Policy and Research department, used here under their copyright. Hand-writing a reference for 100 pages would mean guessing, so it records only elapsed time, output length and OCR routing — each parser running once, the slowest taking about 54 seconds. Original at nseindia.com.
Both leave those axes empty, not zero. A parser never tested on an axis has not failed it.
What this does not measure
No handwriting, encrypted PDFs, photographed pages or non-English text. The scanned fixture is a clean synthetic image — the easiest scan there is — so it proves the OCR axis exists without measuring OCR quality. Hosted Hyperbots OCR is excluded from this free benchmark.
The live Q&A uses one chunker, one embedding model and the same prompt across parsers, so only the parsed text differs. The answer model can still vary at temperature zero, so a disagreement is a reason to read the evidence, not proof either answer is wrong. Costs use configured rates; storage, egress and failed requests can add charges beyond what is shown.
Run it yourself
npm run benchmark refreshes every result, file hash and downloadable fixture — no credentials needed. The generator and reference answers are versioned with the project.