CargoLint

The operation

A customs broker receives a set of documents for a shipment: a commercial invoice, a packing list, sometimes a certificate of origin, sometimes a bill of lading. Somebody reads them, works out what each line item actually is, and assigns a Harmonized System code to it. The code sets the duty, so the whole economics of the shipment depends on a judgement a person makes from a paragraph of product description.

It is manual, so it scales by adding people. And because the judgement is made from a paragraph of product description, the places where the nomenclature is subtle are the places where two careful readers can reach different answers from the same document. Those are the cases the system is built to find and to refuse.

The constraints that shaped the design

Solo build. Every hour on infrastructure was an hour off the classification pipeline, so the architecture leans on managed services.

It is a compliance workflow. A wrong answer delivered with a confident score costs more than a refusal, because a reviewer has no signal to catch it. That single constraint set the design.

Multi-tenant from the first line, with SOC 2 readiness as a design constraint. The data in the system belongs to the operator’s clients. Adding tenant isolation later is a rewrite.

Architecture, and why each piece is there

Multi-stage retrieval. The nomenclature is a hierarchy, so the pipeline narrows to a chapter family first and resolves within it. The distinctions that decide a 6-digit code are invisible at the top of that hierarchy.

Schema-enforced structured outputs, because a free-text answer cannot be validated before it reaches a compliance workflow. If the shape of the response is guaranteed, the downstream code only has to handle the decision.

A confidence score on every field. Extraction and classification fail on different fields and for different reasons. One score for the whole document leaves a reviewer with nothing to aim at.

Provenance back to source text for every extracted field. A reviewer who cannot see where a value came from has to redo the work to check it, and a system that makes checking as expensive as doing has not saved anything.

Chapter-family routing, so the retrieval and the prompting can differ where the nomenclature differs. Textiles and machinery are not the same classification problem and treating them identically costs accuracy in both.

Underneath: ASP.NET Core on Azure App Services, Azure SQL, a Python service for the ML pipeline, Azure OpenAI, and a Vue and TypeScript front end. Timeout handling, retry strategy and structured error telemetry on the model integration, which is where a production AI system actually breaks.

The demo, and the case in it

The centrepiece is a refusal, not a success. The packing list describes a garment as knitted. The certificate of origin describes the same garment as woven. That single distinction decides chapter 61 against chapter 62, and the two source documents disagree with each other.

The system flags the conflict, shows the sentence in each document that produced it, and declines to assign a code. There is no model improvement that resolves this, because the information needed to resolve it is not present. The only correct behaviour is to escalate it to a human with the conflict already isolated.

This is the part of the demo that changes the conversation with a compliance buyer, and it is the part a demo optimised for impressiveness would have cut.

What the evaluation showed

Run 1 of the benchmark scored 96.8% across 1,480 cases, with 98.2% coverage and 98.5% on the answered cases. The weakest slice is trade wording at 88.9%, which is the wording real documents use.

What I would do differently

I built the evaluation harness after the pipeline. Every early architecture comparison was made on impressions, and some of those impressions turned out to be wrong. On the next system the harness comes first.

I calibrated the confidence signal late. Run 1 shows the cost: the 0.75 to 0.90 band scores 93.5% and the 0.60 to 0.75 band scores 97.0%. The score is not monotonic across its own range, and I gated on it before I knew that.

I underestimated the language gap. Run 1 puts trade wording at 88.9% and formal wording at 97.6%. The distance between how a product reads on an invoice and how the nomenclature names it is the dominant source of error. More retrieval work at that boundary would have paid earlier than more prompting downstream.