Benchmark
The HS6 classification benchmark
Accuracy for automated Harmonized System classification, measured on a fixed 1,480 case set, published one page per run with the method attached.
HS6-2022 v1.0 · 1,480 cases
One page per run, every run against the same 1,480 cases. Every figure is computed from the run output. The weak slices are in the tables and they stay there.
What is measured
Every item that crosses a border carries a Harmonized System code. The first six digits are internationally standardised. The national tariff line that sets the duty rate, the trade agreement treatment and any permit requirement is built on top of those six. Get the six wrong and the rest of it is wrong too. That is a misdeclaration, and the exposure sits with the importer.
The nomenclature is legal text. It turns on distinctions that do not appear in the way a product is described on a commercial document: knitted against woven, whether a preparation is put up for retail sale, whether a machine's principal function is the one its name implies. The task is a translation between the language of trade and the language of the tariff schedule. Most of the difficulty sits in that gap, and the numbers below show it.
Why it is published
I went looking for a published accuracy figure in this space with a dataset, a scoring rule, a sample size and a date attached to it. I did not find one. If you are carrying the compliance risk, a bare percentage gives you nothing to work with.
So the set, the scoring rule, the confidence bands and the failures all get published, one page per run, at a URL that does not change. A run that goes backwards gets published with the regression visible. The series is the point, and it starts at one.
Results
One row per run. Accuracy on all counts every declined case as wrong. That is the number to read if you are carrying the risk.
| Run | Date | Set | n | Coverage | Acc. answered | Acc. all | Status |
|---|---|---|---|---|---|---|---|
| v1.0 · gpt-5.4-nano | Aug 14, 2026 | HS6-2022 v1.0 | 1,480 | 98.0% | 97.7% | 95.7% | Published |
| v1.0 · gpt-5-mini | Aug 13, 2026 | HS6-2022 v1.0 | 1,480 | 98.2% | 98.5% | 96.8% | Published |
Method, in short
The set
1,480 cases built from WCO HS 2022 nomenclature. Each case pairs an input description with the 6-digit subheading it resolves to. Half the set is worded close to the official text and half is reworded, because the reworded half is what a commercial document looks like.
What counts as correct
An exact match on all six digits. The right chapter with the wrong subheading scores zero. A two-digit near miss scores zero. A near miss is still a misdeclaration.
Confidence gating
The pipeline returns a confidence per field and declines below the gate. That produces two numbers: accuracy on the cases it answered, and accuracy across all cases with declines counted wrong. The gap between them is the cost of the gate.
Set versioning
The set carries a version. Runs on different set versions are not comparable, so the version sits in the results table.
Get the next run
One email when a run publishes. Nothing else, and no sequence.
Sent by Edison Sinani, 895 Don Mills Rd, Two Morneau Shepell Centre, Suite 900, Toronto, Ontario M3C 1W3, Canada. Every message carries this address and an unsubscribe link, honoured within 48 hours.