If you've ever tried to extract structured data from a bill of lading using a general-purpose OCR service, you already know the punchline: it doesn't work. Not reliably, anyway. And in international trade compliance, "not reliably" is the same as "not at all."
At Atlas Verified, we process thousands of trade documents — bills of lading, organic certificates, phytosanitary certificates, certificates of origin, commercial invoices, packing lists, and dozens more. Early on, we evaluated every major OCR platform on the market. They all failed the same test: give them a document they haven't been specifically trained on, and the output ranges from incomplete to dangerously wrong.
This post explains why we built a custom document processing pipeline, what makes trade documents uniquely difficult, and the engineering principles that guide our approach.
The Document Diversity Problem
Most OCR services are optimized for a narrow set of document types. Google Document AI excels at invoices and receipts. AWS Textract handles structured forms well. Azure Form Recognizer can be trained on custom templates. But international trade generates a staggering variety of paperwork.
In our system alone, we handle over 35 distinct document types. A bill of lading looks nothing like an organic certificate. A phytosanitary certificate from the USDA shares almost no structural similarity with one issued by the EU. An arrival notice from one shipping line may be a structured PDF with clean tables; from another, it's a scanned fax with handwritten annotations.
The critical insight is that each document type has its own schema — a specific set of fields that matter for compliance verification. A bill of lading needs shipper, consignee, notify party, container numbers, vessel name, ports of loading and discharge, and commodity descriptions. An organic certificate needs the certifying agent, operation name, NOP ID, certified products, and effective dates. A certificate of origin needs the origin country declaration, manufacturer details, and harmonized system codes.
Generic OCR tools don't understand these schemas. They extract text. They might identify tables. But they don't know that the string "MEDU4107760" is a container number, or that "NOP ID: 7880315519" is a critical identifier that needs to be verified against the USDA Organic Integrity Database. Without schema awareness, OCR is just expensive text conversion.
Why "Good Enough" Is Dangerous
In most software applications, a 95% accuracy rate is excellent. In trade compliance, it's a liability.
Consider a container number. The standard format (ISO 6346) is four letters followed by seven digits, with the last digit being a check digit. If OCR misreads a single character — a "0" becomes an "O", an "8" becomes a "B" — the container number is invalid. Downstream systems that attempt to track that container will return no results, and the verification chain breaks silently.
Or consider an NOP ID on an organic certificate. This is the identifier that links a certified operation to the USDA's Organic Integrity Database. A misread digit doesn't produce an error — it produces a match against a different operation, or no match at all. Either outcome can lead to a false compliance determination.
The cost of errors compounds through the verification pipeline. A misread shipper name means OFAC sanctions screening runs against the wrong entity. A misread port code means tariff calculations use the wrong duty rate. A misread HS code means the wrong regulatory requirements are applied. Each of these failures is invisible until someone manually reviews the results — which defeats the purpose of automation.
The Case for a Multi-Stage Pipeline
Our solution isn't a single, monolithic OCR model. It's a multi-stage pipeline where each stage has a specific job, and later stages compensate for the limitations of earlier ones.
The pipeline follows a "cheap and fast first" philosophy. The initial stages use lightweight, deterministic tools: direct text extraction from native PDFs, table structure parsing, and traditional OCR for scanned documents. These stages are fast, inexpensive, and produce reliable results for the portions of documents they can handle.
When these stages leave gaps — and they always do — more sophisticated AI-powered extraction fills in. This approach has several advantages over going straight to an expensive AI model:
Cost efficiency. Running a large language model on every page of every document is prohibitively expensive at scale. By extracting what we can cheaply first, the AI layer only needs to handle what the simpler tools missed.
Latency reduction. Users upload documents expecting fast results. The pipeline returns preliminary data quickly from the fast stages while the AI stages work in the background.
Redundancy. When multiple stages extract the same data point, we can cross-reference them. If traditional OCR and AI extraction agree on a container number, confidence is high. If they disagree, we know to flag it for review.
Schema-Aware Extraction
The most important architectural decision we made was to make extraction schema-aware from the start.
When a document enters our pipeline, the first step is classification: what type of document is this? The classifier examines text density, keyword patterns, and structural layout to determine the document type. For ambiguous cases, it escalates to a vision-capable model that examines the actual appearance of the document.
Once classified, the extraction stages know exactly what to look for. A bill of lading extraction focuses on shipping parties, container details, and routing information. An organic certificate extraction focuses on certification scope, NOP identifiers, and validity periods. This focus dramatically improves accuracy because the extraction model isn't guessing what might be important — it's searching for specific, well-defined fields.
Each document type has a formal schema that defines required fields, optional fields, data types, and validation rules. Extracted data is validated against this schema. If required fields are missing or values fail validation, the pipeline runs a targeted repair pass — asking the AI model to look specifically for the missing data, often with a zoomed-in view of the relevant document region.
Confidence Scoring and Graceful Degradation
Not all extractions are created equal. A container number extracted from a clean, native PDF table is high confidence. The same number pulled from a blurry scan of a faxed document is low confidence. Our pipeline tracks confidence at the field level, not just the document level.
This confidence metadata serves two purposes. First, it tells downstream verification systems how much to trust each data point. A high-confidence container number can be automatically tracked; a low-confidence one should be presented to the user for confirmation. Second, it enables intelligent prioritization — if the pipeline has high confidence on all critical fields, the document can proceed to verification immediately. If key fields are low confidence, they're queued for human review.
We also believe strongly in graceful degradation. A pipeline that returns nothing when it can't achieve perfect extraction is less useful than one that returns what it can with honest confidence scores. If we can extract 8 of 10 required fields with high confidence, we deliver those 8 and clearly indicate what's missing. This lets the user fill in the gaps rather than starting from scratch.
What We've Learned
Building a custom document processing pipeline has taught us several lessons that aren't obvious from the outside:
Document classification is harder than extraction. Once you know what type of document you're looking at, extracting the right fields is relatively straightforward. But the classification problem — especially for documents in languages you don't read, or documents that combine elements of multiple types — is genuinely difficult. We've invested as much engineering effort in classification as in extraction itself.
Table extraction is still an unsolved problem. Despite decades of research, reliably extracting structured data from tables in PDFs remains surprisingly hard. Column alignment, merged cells, spanning headers, and inconsistent formatting all break naive approaches. We've found that combining multiple table extraction strategies and cross-referencing their outputs produces far better results than relying on any single method.
The long tail is where the real work is. The first 20 document types cover 80% of the volume. But the remaining types — weight certificates, charge sheets, QMS procedures, residue analyses — each appear infrequently enough that traditional fine-tuning approaches don't work. Schema-aware extraction with well-designed prompts handles this long tail far more gracefully than training specialized models for each type.
Validation is not optional. Early versions of our pipeline had no validation step. The extraction model would occasionally produce structurally valid but semantically wrong output — dates in the future, negative weights, shipper and consignee swapped. Adding schema validation with repair passes caught these issues before they reached the user.
Looking Forward
Document processing in trade compliance is a moving target. New document types appear as regulations change. Existing documents evolve as carriers and agencies update their formats. The pipeline must adapt without requiring a complete retraining cycle every time a shipping line redesigns their bill of lading template.
Our investment in schema-driven architecture pays dividends here. Adding support for a new document type means defining a new schema and writing extraction guidance — not retraining a model from scratch. The multi-stage pipeline architecture means we can swap out or add individual stages without disrupting the rest of the system.
The goal isn't perfect OCR. The goal is reliable, verifiable data extraction that compliance professionals can trust. That requires understanding not just the text on the page, but the meaning behind it — and that's a fundamentally different engineering challenge than generic optical character recognition.
Atlas Verified builds AI-powered verification tools for international trade compliance. Our document processing pipeline handles 35+ document types across the global supply chain.