Lab · technical verification
Token compression reproduction
Verdict
PARTIALLY REPRODUCED
01
The published claims, verbatim
Two public descriptions of the same project differ. Both are recorded exactly; the discrepancy was resolved against the experiment, not in it — the protocol tests the lower coding bound (15%).
“Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.”
“60–95% fewer tokens (for JSON data), 15-20% fewer tokens (for coding agents)”
The repository description says 20% for coding agents; the README says 15–20%. Where a published range and a published point differ, the experiment tested against the lower bound — a deliberately generous reading.
02
Experiment design
The protocol — dataset, baselines, thresholds and verdict rules — was committed and hashed before any measurement was taken. The dataset was constructed by the analyst, not chosen by the project.
Why MINIFIED is the primary baseline: pretty-printed JSON is largely whitespace, and stripping whitespace costs one line of code. A compression tool earns its place by what it saves beyond that. Defaulting the comparison to RAW would make every result look stronger than it is — so this page defaults to MINIFIED everywhere, and shows RAW as the labelled marketing comparison.
0c8aa36e13d554177bf18a57ba263f2fe5c678ea53929c6caa6b778b8469eaf0dataset manifest sha256c16472c28c06194bfc0ffcf57c0acc77363847866835014e6a19b4680f3622bfOne input is withheld from the public repository
The run used all 35 samples and its results are reported unchanged. Thirty-four of them ship in the public repository; one structured-JSON sample is a slice of a raw GitHub profile cache carrying names, locations, bios and contact fields, and redistributing it in bulk would break this project’s own privacy rules. Its metadata and original SHA-256 are published in its place.
03
Results, by category and baseline
Every dot is one sample. Switch the category and the baseline — the default is the decisive one, HEADROOM vs MINIFIED. Negative values mean headroom's output tokenised larger than the baseline.
Per-sample values as a table (12 rows, tokenizer o200k_base)
| Sample | Raw tokens | Minified | Headroom | vs MINIFIED | Probe retention |
|---|---|---|---|---|---|
| json_out_review_queue | 26,649 | 18,178 | 19,068 | -4.90% | 1.0000 |
| json_out_source_yield | 691 | 453 | 453 | 0.00% | 1.0000 |
| json_papers | 2,760 | 1,985 | 1,704 | 14.16% | 1.0000 |
| json_out_holdout_results | 1,352 | 930 | 795 | 14.52% | 1.0000 |
| json_out_negative_controls | 1,695 | 1,194 | 975 | 18.34% | 1.0000 |
| json_out_graph | 83,774 | 61,988 | 45,220 | 27.05% | 1.0000 |
| json_out_signals | 129,035 | 98,190 | 68,948 | 29.78% | 1.0000 |
| json_out_attention_vs_construction | 3,639 | 2,386 | 1,225 | 48.66% | 1.0000 |
| json_repos_40 | 14,774 | 10,290 | 3,643 | 64.60% | 1.0000 |
| json_repos_all | 37,094 | 25,794 | 3,781 | 85.34% | 1.0000 |
| json_contributors | 21,281 | 11,953 | 1,603 | 86.59% | 1.0000 |
| json_users_120 | 13,414 | 9,036 | 1,188 | 86.85% | 1.0000 |
Canonical distributions (o200k_base, pre-registered quantiles)
| Category | n | vs raw — median (p25 · p75) | vs minified — median (p25 · p75) | minify alone vs raw | retention |
|---|---|---|---|---|---|
| Structured JSON | 12 | 46.30% (39.00 · 86.19) | 28.41% (14.25 · 80.16) | 30.84% | 1.0000 |
| Coding context | 12 | 0.00% (0.00 · 0.00) | -8.00% (-9.52 · -3.68) | 7.40% | 1.0000 |
| Agent context | 11 | 0.00% (0.00 · 28.94) | -8.67% (-10.57 · -0.11) | 8.69% | 1.0000 |
04
Pre-registered claims, scored
| Claim | Threshold | Measured | Result |
|---|---|---|---|
| CLAIM-AReduces tokens relative to raw input (structured JSON) | > 0% vs raw (structured_json) | 46.30% | ● SUPPORTED |
| CLAIM-BReduction exceeds trivial minification (structured JSON) | >= 25.0% vs minified | 28.41% | ● SUPPORTED |
| CLAIM-CTask-relevant information survives — “same answers” | >= 0.95 probe retention | 1.0000 | ● SUPPORTED |
| CLAIM-DJSON reduction reaches the published 60–95% band | 60.0-95.0% vs raw | 46.30% | ✕ NOT SUPPORTED |
| CLAIM-ECoding-agent reduction reaches the published 15–20% band | >= 15.0% vs raw | 0.00% | ✕ NOT SUPPORTED |
05
The baseline is part of the claim
Step through the three comparisons. The measured data never changes — the same tokens, the same samples, the same run. What changes is the zero point.
The published comparison. Measured against pretty-printed JSON, the median saving is large.
- vs raw
- 46.30%
- minification alone
- 30.84%
- vs minified
- 28.41%
For structured JSON, headroom saves a median 46.30% against pretty-printed input — but trivial whitespace minification alone saves 30.84% (median) on the same samples. Measured against minified input, headroom’s own contribution is a median 28.41%.
That remaining 28.41% is real, lossless, and worth having. But a majority of the headline number is supplied by a baseline choice, not by the compressor — which is why any compression claim is unevaluable until you know what it was measured against. The baseline is part of the claim.
06
The coding claim, and three attempts to rescue it
All 12 coding samples produced 0.00% savings versus raw — at the minimum, median and maximum. Before concluding anything, the experiment tried to prove itself wrong three ways.
SUPP-1
The 0% coding result is an artifact of presenting a single recent tool output; real savings come from compressing OLDER conversation turns.
FALSIFIED. Every message was routed to a protected class. Multi-turn framing does not change the result.
SUPP-2
Context pressure triggers compression that a large model_limit does not.
FALSIFIED. Tight context pressure produced no additional compression.
SUPP-3
Code compression requires the optional [code] extra, which the base install omits.
FALSIFIED for the library path. The extra installs, but the router still protects recent code.
The mechanism is visible in the router traces: recent code and user messages are routed to a protected class and passed through untouched. On this benchmark, the published coding-agent savings did not materialise under the library’s default entry point — including with the [code] extra installed, under context pressure, and in multi-turn framing. Versus minified, the coding median was -8.00% — negative, because headroom’s structural wrapping adds tokens it cannot recover on content it declines to compress.
07
What this result is — and is not
Headroom is real engineering
The structured-data result is strong
The coding claim did not reproduce here
This benchmark is limited