Audit the decisions made by the person and AI together, alongside human-only and AI-only baselines. Record which errors the person corrects, which new errors the collaboration introduces, and what happens when a case needs escalation. An accurate model does not by itself establish an effective workflow.
This is an evaluation guide with an illustrative worksheet, not a certification standard or a study reporting observed results. For the research question about financial auditors, see Human-AI collaboration in the audit.
1. Define the decision and responsibility#
Write a one-sentence decision, such as “Does this document require specialist follow-up?” Define the evidence used to judge the answer, who makes the final decision, who may override the AI, and who receives unresolved cases. Agree on the relative consequences of a missed issue and an unnecessary referral before looking at results.
The human–AI interaction appendix of NIST AI RMF 1.0 explains why human roles and responsibilities need explicit definition and why team outcomes can differ from either participant alone. The workflow and numerical example below are our own illustration of those concerns.
2. Compare three conditions on comparable cases#
| Condition | What to record | What it tells you |
|---|---|---|
| Human only | Decision, evidence, time and reviewer experience before exposure to AI advice | The existing workflow's baseline |
| AI only | Frozen model/version, prompt or settings, output and abstentions | What the system contributes in isolation |
| Human with AI | AI advice shown, final decision, changes, time and escalations | The outcome people actually receive |
Use a held-out case set with an independently adjudicated reference answer. Keep adjudicators unaware of the experimental condition where feasible. Match the information and time available to reviewers, document unavoidable differences, and allocate reviewers across conditions. Reusing the same case with the same person can create a memory effect; separate reviewers or counterbalance presentation and account for that dependence in analysis. Keep training and tuning cases out of the final evaluation.
Cover routine cases, difficult cases, incorrect AI advice and cases where the tool abstains. Report natural-prevalence results separately from any deliberately enriched challenge set. Keep unresolved cases visible rather than counting them as successes.
3. Measure outcomes and interactions#
For a binary follow-up decision, let TP be a required follow-up correctly requested, FN a required follow-up missed, FP an unnecessary follow-up and TN a correctly cleared case.
| Measure | Calculation | Interpretation |
|---|---|---|
| Accuracy | (TP + TN) / all resolved cases | Overall correctness; may hide rare but costly misses |
| Recall | TP / (TP + FN) | Share of required follow-ups found |
| False-positive rate | FP / (FP + TN) | Share of clear cases unnecessarily referred |
| Precision | TP / (TP + FP) | Share of requested follow-ups that were required |
| Override rate | Final decisions differing from AI / cases with AI advice | How often the person changes the recommendation |
| Correction yield | AI errors corrected / overrides | Whether disagreements improve the outcome |
| Harmful overrides | Correct AI answers changed to incorrect answers / overrides | Errors introduced during review |
| Escalation completion | Escalations resolved within the agreed time / escalations due | Whether the fallback works in practice |
Report numerator and denominator. A zero denominator means “not applicable,” not 0%. For abstentions, report coverage as the fraction of cases with a decision, plus what happened to the remainder. A low override rate alone cannot distinguish trust from uncritical acceptance; review the underlying cases.
Compare error types by task difficulty and relevant user groups, with sample sizes and uncertainty. Record workload, review time, explanations and reviewer feedback alongside accuracy. NIST's Measure Playbook supports selecting context-appropriate measures, documenting what cannot be measured, defining acceptable limits and checking performance after deployment. It does not prescribe the example thresholds here.
4. Worked example: 200 synthetic cases#
Suppose 40 of 200 documents require follow-up. The other 160 do not. These invented decisions illustrate the arithmetic; they are not performance evidence for any product. Download the complete example CSV to reproduce the counts.
| Condition | TP | FN | FP | TN | Accuracy | Recall | False-positive rate |
|---|---|---|---|---|---|---|---|
| Human only | 28 | 12 | 20 | 140 | 84% | 70% | 12.5% |
| AI only | 32 | 8 | 24 | 136 | 84% | 80% | 15% |
| Human with AI | 36 | 4 | 16 | 144 | 90% | 90% | 10% |
For the joint condition, accuracy is (36 + 144) / 200 = 90%, recall is 36 / 40 = 90%, and precision is 36 / 52 ≈ 69.2%. Accuracy improves by 6 percentage points over each baseline. The team still misses four cases.
The CSV has 20 decisions that differ from AI advice: 16 correct AI mistakes and four change a correct answer into an error. Override rate is 20 / 200 = 10%, correction yield is 16 / 20 = 80%, and harmful overrides are 4 / 20 = 20%. Of the AI's 32 errors, 16 remain in the final decisions; reviewing only the overrides would miss these accepted errors.
For an illustrative cost of five units per miss and one per unnecessary referral, loss is 5 × FN + FP: 80 for human only, 64 for AI only and 36 for collaboration. These weights are assumptions to replace with the task owner's agreed values. They do not represent a universal ranking of harms.
The example supplies no evidence about review time, subgroup performance, escalation completion or statistical significance. In a real evaluation, examine paired case-level differences, dependence between reviewers, confidence intervals and the costs of running the workflow before deciding whether the gain is reliable.
5. Keep an audit trail and decide what to change#
| Log field | Why it is needed |
|---|---|
| Case ID, task, sampling stratum, reference answer and adjudication status | Reconstruct denominators and disputed labels |
| Model/version, prompt or settings, AI output, timestamp | Reproduce what the reviewer saw |
| Reviewer role, advice shown, final answer and reason for changing or retaining it | Inspect reliance and accountability |
| Review duration, escalation owner, deadline and final resolution | Detect delays and failed handoffs |
| Error severity, affected group where relevant, remedy and re-test date | Connect the finding to an action |
Retain only the evidence needed for the evaluation under the organization's access and retention rules. Keep personal identifiers out of exported comparison worksheets.
Before the evaluation, choose acceptable error and workload limits, assign an owner to each failed limit, and specify when to revert to the previous workflow or require additional review. For example, an escalation that has no owner is a process defect even when aggregate accuracy is high. Repeat the same evaluation after a model, prompt, interface or reviewer-training change, and monitor for new types of failure during use.
Related research#
The financial-audit research idea asks whether AI changes audit outcomes and responsibility. This guide supplies a way to design a local workflow evaluation; it does not establish the causal effects proposed in that research note.