Testing AI document processing: an acceptance template

Specify document types, required fields, permitted actions and the authoritative destination. Structured invoice data should be validated through the intended data route; running OCR over the visual PDF is not automatically useful. Scans need a separate assessment of recognition and its errors.

Create an approved sample covering ordinary and difficult cases. Authorized real documents may be needed later; anonymized or synthetic examples can provide a starting point. Record their origin and modifications. Establish correct expected values independently of the system being tested.

A small test set is a starting point, not a statistical guarantee of future performance. Extend it when new suppliers, formats or failures appear.

Three measures instead of one score

Three measures describe a trial of AI document processing better than one score:

Field errors: Compare each field with its expected value. Separate missing, wrong and invented values. A correct total does not cancel out an incorrect bank account.

Undetected errors: Establish whether an incorrect critical field passed the agreed control. Correctly referring a difficult case for review is different from incorrectly approving it.

Remaining work: Measure review, correction, questions, retries and routine administration. Compare the same document types with the previous process. Processing speed is not the same as time released for your team.

For example, 20 synthetic documents with five evaluated fields each contain 100 field values. Three incorrect values mean 3% in that particular set. This does not mean “97% of all invoices work”, and says little about document types not yet encountered.

Measure precision and completeness separately

Precision and recall. Google Document AI describes evaluation against labelled test documents using precision and recall. Precision asks what share of returned values is correct. Recall asks what share of expected values was correctly found. A system can return few wrong values while still omitting information the workflow needs. Google evaluation guidance, checked on 26 September 2026.

Per critical field. Report each critical field separately. Invoice number, amount, currency and supplier have different consequences when wrong. For line items, also count missing and duplicate rows. Decide beforehand whether differences such as whitespace are acceptable; do not change the matching rule afterwards to make results look better.

Synthetic example. In a synthetic example, 100 documents contain a purchase reference. The system returns 90 references, of which 85 are correct. Precision is 85/90, approximately 94.4 percent, while recall is 85/100, or 85 percent. Both describe only this field and this test set. The five incorrect outputs and 15 references not correctly recovered call for different improvements.

What a confidence score means. A model score of 0.95 does not guarantee that 95 percent of your business cases are correct. Microsoft describes confidence as estimated certainty for particular predictions and explains how to assess it. Microsoft accuracy and confidence guidance.

Count errors per range. Group outputs into confidence ranges and count actual mistakes in each. Evaluate critical fields separately. A legible document assigned to the wrong company may have high recognition confidence and still be operationally wrong. Review routing should therefore combine confidence, deterministic rules and business context.

Thresholds per field. A threshold chosen to maximise a general score may not match your priorities. If a missed payment reference merely requires review while a wrong bank detail has greater consequences, the same automatic acceptance rule is inappropriate for both. Define the permitted action at field and transaction level.

Record versions. Record model version, prompt, rules, mapping and destination configuration. When one changes, compare the new variant on the same frozen test set. Add newly discovered failures without replacing the original comparison. This reveals regressions as well as improvements.

Separate preparation and approval. Separate test preparation and approval where practical. The person changing a mapping should not be the sole authority deciding whether an ambiguous expected value should be revised afterwards. Keep disputed cases visible and let the responsible specialist decide.

Acceptance note. End with an acceptance note covering scope, cases actually executed, known limitations, permitted automation and the next review. This is more useful than a general statement that the AI works because it tells operators exactly what the evidence allows them to do.

Agree thresholds before evaluating results. A possible starting point is no undetected critical error in the agreed sample, all defined failure routes demonstrated, and a documented manual fallback. The responsible domain owner must decide other acceptable error rates based on consequences and effort.

Zero observed errors still does not prove error-free operation. Introduce the workflow gradually, sample outputs and rerun critical tests after changes to recognition, rules or interfaces. Record unresolved issues with owners and dates instead of applying a blanket “passed” label.

A supplier handover for AI document processing should contain:

• the sample
• expected values
• results
• unresolved deviations
• approval rules
• recovery instructions
• where originals, logs and configuration reside
• who owns operation

See AI document processing for the service scope. BMD users should begin with existing product capabilities. The approval guide explains what makes human review meaningful.