Intelligent Document Processing: How to Evaluate It

A vendor demo of document processing almost always works, because it runs on documents the vendor has seen before. What you need to know is what happens to the invoice from the supplier who still faxes, the statement that lost page four, and the KYC pack with a stamp across the address. This guide is a test plan for that: build a set from your own documents, measure every field, check whether the confidence scores mean anything, and score the tools on what you find.

Three generations, and what each hands you

OCR turns an image into characters. Template-based IDP maps named fields onto layouts someone configured. Agentic extraction finds fields by context, checks them against your data and says how sure it is about each one. All three are sold as intelligent document processing, so start by working out which one is in front of you.

The test plan below works for all three. It is deliberately harder on the third, because a tool that claims to reason about your documents should be asked to show it.

Decide what you are evaluating

Write down three things before you contact a vendor. First, the document types and their monthly volumes: supplier invoices, receipts, bank statements, purchase orders, KYC packs, and anything particular to your business. Second, the fields you need from each, and only those. A list of forty invoice fields when the downstream step uses twelve makes every tool look worse and teaches you nothing. Third, what happens to the data next: posted to an ERP, matched in a payables workflow, filed for audit. That last answer decides which errors are expensive.

For each field, note what a wrong value costs. A wrong remarks line costs nothing. A wrong total, GSTIN or bank account number costs money or a compliance problem. Spend the evaluation's attention on those high-cost fields.

Build the test set from your own documents

Pull the documents from a recent month as they arrived, rather than from a folder of good examples. For each one, write down the correct value of every field on your list. That answer key is the most useful thing the evaluation produces, and it stays with you. Send vendors the documents only, then compare what comes back.

Seventy-odd documents is enough to see patterns and few enough that one person can write the answer key. Mask what you must before anything leaves your premises, and agree in writing what the vendor does with the files and for how long. A masked field should come back blank or masked, which is the correct result.

Measure per field, not per document

A single accuracy percentage tells you very little. It depends on what was counted (fields or documents, every field or only the easy ones, before or after review), and it folds together four outcomes with very different costs.

Put every field on every document into one of those four boxes, then read the results field by field. The box to watch is the silent error: a wrong value the tool was confident about and would have posted. On high-cost fields that count should be zero in your set, and a tool that produces silent errors on totals, tax IDs or bank details should not pass. Review cost matters too, but it is a staffing question, and it falls as thresholds are tuned.

Read the results by slice as well. A tool that is excellent on your top suppliers and poor on the long tail looks fine in aggregate and fails on the documents that cause the work.

Test the confidence scores themselves

Confidence is only useful if it means something, and you can check that. Sort every field in your results by its score and look at where the wrong values sit. If the scores are meaningful, the errors cluster at the bottom. If errors are spread through the high scores, no threshold can protect you, wherever it is set.

Then ask where the threshold is set and by whom. It should be per field and per document type, set by you, and changeable without a software release. A single document-level score hides the field that matters: an invoice can score well overall while its total is the one value that is wrong. The DocPro page explains how confidence is scored per field and how thresholds are set by field and by document type.

Validation and review

Reading is half the job. Ask each tool to check what it read: line totals against the invoice total, tax against rate and taxable value, the GSTIN format, the vendor against your master, the running balance on a statement from page to page. The known-bad documents in your set exist to test exactly this. A tool that returns a duplicate invoice, or a statement with a missing page, looking clean has failed the most useful test in the plan.

Then sit someone from your team in front of the review screen with the held fields. A good screen shows the field, the value read, the region of the page it came from and the reason it was held, so the reviewer can settle it without opening the original. Time a dozen decisions. If the reviewer keeps opening the full document, review will cost more than anyone estimated. The human-in-the-loop guide covers how to design the queue around those decisions.

Red flags in vendor demos

• The demo runs only on the vendor's sample documents, and using yours is deferred until after the contract is signed.
• Accuracy is quoted as one number, with no statement of what was counted or on which documents.
• Confidence is shown per document, or not shown at all.
• A new layout needs a template, a training cycle or a services ticket before it can be read.
• Wrong values come back with high confidence, and the answer is that the model will improve with use.
• Validation is described as available but is not run on your known-bad documents.
• The output is a spreadsheet or a screen, when your systems need structured data with the scores and checks attached.
• Questions about where documents are processed, and for how long they are kept, get a general answer.

None of these proves a tool is bad. Each one is a reason to run the test plan before believing the demo.

A scoring checklist

Run the must-pass group first, on your set. Score the second group only for the tools that pass, one to five per line, with the people who will run the process doing the scoring. Weight the lines by how much of your work they touch. A team that adds a document type every quarter should care more about change effort than one with a single, stable invoice flow.

What document processing will not do

Document processing reads what is printed. It cannot supply a cost centre that is not on the invoice, judge whether an over-limit expense is acceptable, or decide whether a KYC applicant should be accepted. Those need a lookup, a rule or a person, and a vendor who suggests otherwise is describing a different product. On badly degraded scans, expect lower confidence and some flagged blanks. The right behaviour on a field that cannot be read is to flag it, and a tool that guesses instead will eventually guess a total.

How to test this on your own documents

1. List the document types, the fields you need and the cost of a wrong value in each.
2. Build the set from a recent month, with an answer key you keep.
3. Send each tool the documents only.
4. Put every field into one of the four outcomes and read the results by field and by slice.
5. Sort by confidence and check where the errors sit.
6. Time a reviewer on the held fields.
7. Score the must-pass group, then the rest.

You can see part of this today. The public DocPro test reads one invoice or UPI receipt and shows the fields beside the page. It does not display confidence or run validation, so treat it as a first look. For the full plan on your own set, with confidence, validation and review, request a DocPro demo. Intelligent Document Processing describes how DocPro classifies, reads, validates and routes each document.