Six planned parsers, replaced by one layer, after reading 200 extractions that had already failed.
Lead Product Manager - Owned Processing architecture, model routing policy, evaluation framework, reviewer workflows.
Led delivery across 2 engineering pods, ML ops engineers, and a 50-person review ops team.
The Setup
In document extraction, the demo always works. The long tail is what kills you.
Enterprise documents are not one problem. They are layouts, tables, handwriting, scans, images, and mixed modalities, arriving in formats that vary by customer, by vendor, and by whichever version of a template someone last edited. A system that handles the ten formats you tested on will fail on the eleventh, and the eleventh arrives the week after you ship.
The standard answer is to build a parser per format. It works, and the roadmap had six of them queued.
The problem with that answer is the slope. Every new customer brings variants nobody has seen, so engineering cost scales with document diversity rather than with revenue. Six parsers buys you six formats and commits you to building the seventh, and the twentieth, forever.
The Decision
Before adding to that queue, I went looking for what was actually failing, and found that we had already collected the argument against our own roadmap.
I analyzed 200+ failed extractions. The failures were not format-specific. They clustered around what the system was trying to find rather than what the document looked like, which meant a per-format fix was treating the symptom. Six separate parsers would have solved six instances of one problem.
So I killed all six and replaced them with a schema-driven retrieval architecture: define what a field is and how to recognize it, then let retrieval generalize across layouts instead of hard-coding each one.
Six planned builds, and the case against all of them was sitting in documents we had already failed on.
The consequence mattered more than the architecture. Onboarding a new document variant stopped being an engineering project and became a configuration change.
What It Cost
The parsers were not a bad idea, and I want to be accurate about why.
Per-format parsing was hitting very high accuracy, but only because engineering had spent hundreds of hours hand-fixing each format individually. That effort was real and the results were real. Schema-driven retrieval did not require it, which was the entire point, and it also meant the new system started below what the hand-tuned parsers were already delivering.
So the trade was explicit: give up accuracy on the formats we had already solved, in exchange for accuracy that generalized to the ones we had not seen. It improved through feedback loops rather than through more engineering hours, but it improved from behind.
Defending a system that looks worse on day one is a harder conversation than shipping the thing that looks better immediately. That conversation is only winnable with measurement, which is the real reason the evaluation framework existed.
What Shipped
Pipeline - Ingestion, classification, extraction, validation, exception handling
Retrieval - Schema-driven layer that generalizes across layouts instead of hard-coding them
Routing - Complexity-based model routing to balance accuracy, latency, and cost
The layer that made it trustworthy
Confidence thresholds determining which extractions could pass automatically and which required a person · exception handling for cases the system should not attempt · an evaluation framework on annotated benchmark datasets measuring extraction accuracy, field-level precision, hallucination rate, and confidence calibration · reviewer corrections routed back as system-level quality loops
Outcomes
100K+
Documents Processed (Monthly)
95%+
Extraction Accuracy
70%
Turnaround Time (Lifted)
The eval framework did more than track quality. It guided model selection, prompt iteration, retrieval tuning, and release decisions, which is what let us ship on evidence rather than on the last demo anyone remembered.
New-variant onboarding became a configuration change rather than a multi-week build, so the engineering cost of the product stopped scaling with the diversity of our customers' documents.
What I Know Now
The option that looks best on day one is often the one that scales worst, and in AI products that gap is especially wide, because a hand-tuned system on a known set of inputs will beat a general one every time until the inputs change.
The evaluation framework was never really a quality initiative. It was the thing that made a slower-starting architecture defensible, because without measurement the only available argument was that I believed it would be better later. Evaluation is what turns product judgment into something a team can act on.