Elitez OCR — Proof of Work

01The headline is three numbers, not one

Measured on the frozen v5 test set — 6,406 hand-labelled rows across 427 timesheets, scored by the same harness on every run. Reproduce it: score_both.py gt_v5_recon.jsonl qwen7b mistral24b olmocr2.

73.2%

ALL-5 null-matched

98.2%

row recovery

37.2%

ALL-5 strict

42.3%

strict ceiling

Read these together. Row recovery 98.2% — of every dated row on the sheets, we find and align 98 in 100. ALL-5 null-matched 73.2% — on a recovered row, all five fields agree with the label 73% of the time, giving a correct blank cell the credit it deserves (a rest day IS blank). ALL-5 strict 37.2% looks low only because of the strict ceiling: just 42.3% of rows have all five fields filled in the first place (most rows are rest days or partial). So 37.2% is 88% of the maximum a strict all-five score can reach — the strict number is near its mathematical limit, which is why null-matched is the honest measure of reading.

02Measured their way vs measured our way

Vendors quote character-level accuracy on clean printed text ("99% printed, 90%+ handwritten"). That is a real number — but it answers "did it read the glyphs," not "did it produce a correct payroll row." Here is the same frozen 6,406-row set scored four ways, for the union and each model alone.

Model	char-level (their way)	field-exact	row ALL-5 (our headline)	row recovery
qwen2.5vl:7b — alone	84.8%	85.8%	70.3%	90.3%
olmocr2:7b — alone	80.8%	80.8%	67.7%	85.7%
mistral-small-24b — alone	73.7%	72.1%	54.8%	82.9%
3-model union	91.8%	92.8%	73.2%	98.2%

And the vendor claim is specifically for clear printed text. Split our union’s character-level accuracy by document type:

Document type	Char-level (union)	vs vendor claim
printed_table	97.9%	≈ their "99% printed"
spreadsheet	93.7%	chars read — but wrong columns
handwritten_form	86.5%	≈ their "90%+ handwritten"
whole corpus	91.8%	phone photos, mixed quality

So there is no accuracy gap — there is a metric gap. Measured the vendor’s way, our union reads 97.9% of characters on clean printed pages and 91.8% across the whole messy corpus — squarely at their claimed level. Our headline is lower on purpose: 73.2% is the fraction of rows where all five fields are simultaneously correct and in the right columns — the only thing that actually protects a paycheck. The spreadsheet row proves the point: 93.7% of characters read correctly, yet only 4.4% of rows usable — every glyph right, in the wrong column. A character score cannot see that error; ours is built to.

One more read of the table: olmocr2 alone scores 80.8% char-level on our corpus, not 99% — because our timesheets are nothing like the clean documents a vendor benchmarks on. No single model is at its brochure number here; the union is what recovers the difference (+7pt char, +8pt recovery over the best single model).

03Accuracy by field — the component breakdown

The 3-model union, per field. ■ strict (must match a real value) ■ null-matched (a correct blank counts).

Field	Strict	Null-matched	What limits it
date	98.2%	98.2%	near-solved; the weekday cross-check pins it
time_in	67.5%	90.8%	handwriting; `:` misread as `.`/`;`
time_out	65.0%	88.5%	as above
break_minutes	53.2%	87.0%	often unwritten; minutes vs hours ambiguity
total_hours	49.1%	84.6%	the weakest field — recomputed & cross-checked
ALL-5 (row)	37.2%	73.2%	every field must agree at once

Where to push: date is effectively solved (98.2%); the gains left are in total_hours (84.6%) and break_minutes (87.0%) — which is exactly why the pipeline recomputes hours from in/out/break rather than trusting the written total, and flags the disagreements. Predicted summation consistency is 75.2% (2,591/3,444 rows) — a built-in error detector.

04What the 3-model union buys over one model

Same test set, same harness — single best model vs the union.

Configuration	date	ALL-5 null-matched	row recovery
qwen2.5vl:7b — single	90.3%	70.3%	90.3%
3-model union (+ mistral-24b + olmocr2)	98.2%	73.2%	98.2%
gain	+7.9pt	+2.9pt	+7.9pt

Diminishing returns are real and measured — more models HURT. A 4th model (gemma3) drops the union to 72.3%; a majority-vote combiner instead of first-non-null drops it to 70.5%. The 3-model union is saturated: the next gain does not come from more models.

05Accuracy by document type — where the real weakness is

The shipped pipeline scored per template (140-file frozen set). "Conditional" = accuracy on the rows we recovered.

Template	Files	Rows	ALL-5	Recovery	Conditional
printed_table	19	353	64.9%	77.9%	83.3%
handwritten_form	112	1,118	54.2%	88.6%	61.2%
punch_card	6	79	22.8%	38.0%	60.0%
spreadsheet	3	68	4.4%	100.0%	4.4%

The counter-intuitive finding: handwriting is not the weak family — it is mid-pack (54.2%) with the best row recovery. The catastrophic cases are spreadsheets (finds every row, gets every field wrong → a column-assignment problem, not a reading one) and punch cards (38% recovery → their AM/PM/OT punch pairs don't fit a single in/out slot). These are distinct, fixable failure modes — and they need dedicated handling, not better handwriting OCR. (spreadsheet n=3, punch n=6 — directional, not settled.)

06The progression — how 27% became 73%

Every point is a measured run on the frozen harness. The wins came from the pipeline, not from bigger models.

17 Aug~27% ALL-5 — first honest 5-field harness (N=610). Baseline established, two measurement bugs found in the process.
18 Aug31.9 → 34.2% — weekday-constrained period inference: the printed day column resolves ambiguous dates. Segmentation pipeline tried and rejected (8.5%).
22 Augrecovery 77 → 86% — row-level UNION across models clears the row-recovery bar; the bottleneck flips from reading glyphs to finding rows.
23 Aug73.1% null-matched — consensus combiner + weekday-constrained inference break the union's saturation.
24 Aug+10.8pt ALL-5 — cross-model abstention: models INVENT values more than they misread them, so refusing a low-agreement read is the single biggest lever.
25 Aug70.3% single-model — v5 frozen set (6,406 rows). The weekday is confirmed a 97.9%-agreement error-correcting code over 19,133 rows.
3 Sep73.2% union — 3-model union on 6,406 rows: 98.2% recovery, 37.2% strict. Current best; the metric has plateaued for models/voting.

The pipeline journey on the core metric was 27.4% → 74.4%, and all of it came from pipeline work — period inference, row union, consensus, abstention — and none from fine-tuning.

07The LoRA record — five clean losses, and exactly why

Fine-tuning was tested seriously, five times, and reported honestly each time.

Attempt	Base / pool	Result
v1 (#31)	olmOCR-2-7B, 33 files	below zero-shot	loss
v2 (#33/35)	Qwen2.5-VL-7B, 37 files	zero measurable gain	loss
v3b (#57)	schema-aligned, 42 files / 466 rows	date 28.7% vs pipeline 93.6%; 51 hallucinated rows	loss
v6 (#115)	2,606 scans (14× prior)	12.2% exact / 18.7% coverage	loss

The root cause is a data problem, not a model problem. Our inference prompt asks the model to emit a row for every date line, including blank rest days — and on this corpus most rows genuinely ARE blank (only ~1 in 11 rows carries a total). Fine-tuning on that distribution teaches the adapter to "predict null", which it then does even on populated rows. The all-rows prompt is right for inference (it drove recovery to 98%) and actively harmful as a training target at this pool size. Fine-tuning also can't beat a strong generalist at this scale. LoRA is parked — not abandoned — until the three conditions in the roadmap are met.

08The roadmap — how we systematically improve, from here

Grounded in what the 115 measured experiments actually showed. Ordered by expected return.

Recursive learning — the accretive loop (highest ROI, already half-built)

Lever · live
Turn HR corrections into ground truth.
Half of all rows already route to HR for review (the other half auto-clear at HIGH confidence). Every correction is a new labelled example. Capturing them grows the clean labelled pool from hundreds toward the 10k+ that changes what fine-tuning can do — with zero extra labelling cost.
evidence: 50% auto-clear / 50% routed; the de-identification fix (#54) unblocked using real corpus at scale.
Lever · build
Confidence-gate what humans see.
Weekday-vs-date self-consistency already separates a 77.3%-correct band from a 46.1% band at zero cost. Send only the low-confidence band to HR; auto-clear the rest. Every experiment then only spends human time where it moves the number.
evidence: #104 self-consistency signal; #103 weekday agrees with date 97.9% over 19,133 rows.

Pipeline ML — the proven levers with gains left

Lever · proven
Push abstention further.
Cross-model abstention was the single biggest gain (+10.8pt) because the dominant error is invention, not misreading. Extend it per-field (it helps time fields most) and feed the abstain signal into the confidence gate.
evidence: #72 error taxonomy; #73 +10.8pt.
Lever · build
Dedicated handlers for the two catastrophic templates.
Spreadsheets (4.4% — column misassignment) and punch cards (38% recovery — AM/PM/OT pairs vs one slot) will not improve with better OCR. A spreadsheet column-mapper and a punch-card pair-schema are targeted, high-yield fixes on their own rows.
evidence: #76/#77 per-template scores.
Lever · proven
Exploit the weekday error-correcting code + arithmetic checks.
Auto-correct the date field against the printed weekday (97.9% agreement), and use the recomputed-vs-stated hours disagreement (summation consistency 75.2%) as a precise, per-row flag instead of a blanket one.
evidence: #103; §02 summation consistency.

LoRA / fine-tuning — the conditions that would make it win

Contingent
Three things must change before LoRA is retried.
(1) Scale — an order-of-magnitude larger clean pool (10k+ rows), now feasible from the 52k corpus + HR corrections. (2) Balanced sampling — upweight populated rows so the training target isn't null-dominated (the exact cause of every loss so far). (3) Template-specific adapters — train narrow adapters for the weak families (spreadsheet column-assignment, punch-card pairs) and the weak fields (break/total), where a specialist can beat a generalist, instead of a general adapter that competes with a strong zero-shot model and loses.
evidence: #57 root cause (null-dominated targets); #115 scale alone (2,606 files) still lost — so scale WITHOUT balancing is not enough.

What NOT to spend on (measured dead ends)

Keep doing

3-model union (first-non-null)
Cross-model abstention
Weekday/date cross-checks
Recompute hours, flag disagreements
HR corrections → labelled pool

Stop / don't start

A 4th model — gemma3 drops it to 72.3%
Majority-vote combiner — 70.5%
General LoRA at current scale — 5 losses
num_ctx sweeps — null result
Chasing the strict ALL-5 number — near its 42.3% ceiling

09Live experiment — should the union change?

Complete · 6 Sep A standing question: with new open models shipping monthly, should we swap the weakest member of the union, or expand to a 4th/5th model? We survey Hugging Face + the Ollama registry and bench the strongest candidates on the same frozen harness — apples-to-apples, on the fleet, no cloud.

The five candidates under test

Model	Source	Why it's a candidate
qwen3-vl:8b	Alibaba (late 2025)	Newest Qwen VL — replaced qwen2.5-vl at the top; strongest CJK/multilingual. Direct swap candidate.
minicpm-v:8b	OpenBMB	MiniCPM-V 4.5 — widely reported best open document-OCR at 8B.
granite3.2-vision:2b	IBM	Built specifically for tables/charts/document extraction — targets our spreadsheet weakness.
llama3.2-vision:11b	Meta	Strong general VLM — added for error diversity, not raw OCR.
moondream:v2	Moondream	2GB tiny model — the cheap floor; is a small model good enough on printed forms?

The method (identical to every number on this page)

Each candidate runs the same PROMPT_V6 on a stratified small batch of the frozen v5 set, on the free fleet box (monster4), scored the same four ways. Then we score union variants — the current three, each candidate added as a 4th, and each candidate swapped for the current weakest member (mistral-24b). A candidate only earns a place if it moves row recovery or null-matched on held-out data — the bar every past change had to clear, and the bar that has rejected a 4th model and every LoRA so far.

Baseline on this batch (the bar to beat)

Config	char-level	field-exact	row ALL-5	recovery
qwen2.5vl:7b	79.4%	79.0%	59.4%	82.5%
olmocr2:7b	73.6%	71.6%	56.7%	79.1%
mistral-24b (weakest — swap target)	73.2%	69.8%	47.8%	78.4%
CURRENT union	87.8%	87.0%	57.9%	93.8%

Measured results — single-model (30-file stratified slice)

Candidate	char	field	ALL-5	recovery	parsed	read this as
minicpm-v:8b	55.7%	52.3%	16.8%	63.5%	28/30	parsed cleanly — genuinely weaker than any current member
qwen3-vl:8b	21.6%	23.5%	17.8%	19.2%	29/30	artifact, not quality — default thinking-mode (5,075s / 30 imgs) + JSON mismatch
moondream:v2	0.0%	0.0%	0.0%	0.2%	7/30	artifact — output format incompatible with the strict JSON harness
granite3.2-vision:2b	0.2%	0.1%	0.0%	0.2%	1/30	artifact — output format incompatible
llama3.2-vision:11b	0.0%	0.0%	0.0%	0.0%	0/30	artifact — output format incompatible

Four of five produced output our frozen PROMPT_V6 + strict […]-JSON parser could not read — so their near-zero scores measure harness compatibility, not OCR ability. They are reported here honestly as artifacts, and queued for a fair re-test with per-model output adapters (and thinking-mode off for qwen3-vl). Only minicpm-v is a true, clean read — and it lost to every current member.

Measured results — union variants (the decision)

Config	char	field	ALL-5	recovery
CURRENT union (qwen2.5vl + mistral + olmocr)	87.8%	87.0%	57.9%	93.8%
ADD any candidate → 4-model	87.8%	87.0%	57.9%	93.8%
SWAP mistral → minicpm-v	88.2%	88.2%	54.8%	93.8%
SWAP mistral → (format-failed candidate)	84.5%	85.1%	60.1%	88.5%

Verdict: keep the three. No swap, no fourth model. Every ADD left the union unchanged — a first-non-null union is already saturated on recovery by three members, so a fourth contributes nothing unless it beats all three on a field. No candidate did. The lone real signal: dropping mistral lifts ALL-5 to 60.1% but costs 5.3 pts of recovery (93.8→88.5) — proof that mistral earns its seat on row recall, not per-field accuracy. Open follow-up: re-bench the four format-incompatible models with per-model output adapters before any final verdict — a fair test the strict shared harness did not give them. (30-file slice; subset baselines run a few points below the full-corpus 91.8%/73.2% headline — the model-vs-model comparison is what counts.)

10Why the number keeps moving

Accuracy is not frozen at deployment. Rows flagged for review — a low agreement, a weekday that disagrees with its date, a recomputed total that contradicts the written one — are the exact signal for the next improvement, and each human correction becomes a new labelled example. The metric is re-measured on the same frozen 6,406-row set after every change, and only a change that moves null-matched ships. The galleries are the audit trail: every extracted row is drawn on the sheet it came from, so any number here is traceable to its pixels.