A pilot is working when it produces acceptable results within the agreed scope and improves the work without hiding errors or shifting the burden to reviewers. Compare similar cases before and during the trial, include supervision and exceptions, and agree in advance which evidence supports continuing, correcting or stopping the project.
Decide which question the pilot must answer
“The demonstration is convincing” and “this process works for us” are different conclusions. A trial may show that a step is technically possible without establishing how much work remains with the team or how difficult cases are handled. Write down the decision you want to make: continue with the same assignment, correct part of the workflow or suspend it.
First define a correctly completed case: the expected output is available, sources can be checked, reviews are finished and the final status is confirmed. An item processed by software is not automatically an accepted deliverable. Agree who assesses the outcome, choosing someone who understands the work and can challenge the result.
Documenting test cases, metrics and assessment tools is consistent with MEASURE 2.1 in the NIST AI RMF 1.0. The sheet below is a practical proposal, not a certification test.
Build a baseline that makes the comparison fair
Observe the existing process using the same starting and finishing points as the trial. Record case numbers, types and complexity: a week of standard requests does not represent a period containing disputes and incomplete records. If you cannot keep the same mix, separate the groups and explain the difference rather than comparing two averages alone.
Distinguish a person’s active working time, waiting time and the total elapsed time from request to handover. A case may take only a few minutes of work yet wait for days because a source is missing. Reducing the first measure does not establish that the second has improved.
Use the same acceptance criterion and observation point for completed outputs. Show outstanding cases, exclusions and later corrections. A denominator made up solely of successful cases hides unfinished work. The guide to preparing your first project helps establish this boundary before you start collecting figures.
The measurement sheet: data, owner and decision
Complete the criteria with the owner before the trial. These prompts describe how to decide; they do not prescribe universal thresholds. Specify which errors prevent continuation even if average handling time improves.
Scroll the table horizontally to read every column.
| Metric | Source and owner | Continue, correct or stop |
|---|---|---|
| Volume and complexity | Case list showing arrivals, exclusions and outstanding work by type; operational contact. | Continue if the sample covers the assignment. Correct the sample if variations are missing. Do not generalise to unobserved cases. |
| Total human time | Preparation, review, exceptions and supervision recorded without duplication; team co-ordinator. | Continue if the effort is sustainable and supports the objective. Correct steps that shift too much work to reviewers. |
| Waiting and elapsed time | Arrival, blocking, resumption and acceptance timestamps; process owner. | Correct waits with no owner. Stop the affected step if blocked work is presented as completed. |
| First-review acceptance | Initial review outcome and reasons for rejection; reviewer from the relevant function. | Compare with the agreed criterion. If it is not met, correct the process before expanding its scope. |
| Errors and rework | Correction log, severity and time required; owner of process quality. | Correct repeatable defects and test again. Suspend for the critical errors defined in the plan, even if infrequent. |
| Human interventions | Number, reason and outcome of requests for help; exception owner. | Distinguish planned interventions from failures. If the queue cannot be handled sustainably, correct or reduce the assignment. |
| Cost per accepted case | Scoped costs and accepted case count for the same period; Finance with the process owner. | Continue only with a complete comparison. Correct the calculation before an economic decision if costs or open cases are missing. |
A worked example: faster preparation can need more review
The following figures are invented to explain the calculation; they are not Digital Employee results. Consider two comparable sets of 100 cases, each containing 80 ordinary cases and 20 with exceptions. All 100 cases in each set have been accepted after any corrections, and human time is counted only once.
Scroll the table horizontally to read every column.
| Activity | Baseline | Pilot |
|---|---|---|
| Remaining manual preparation | 600 minutes | 100 minutes |
| Output review | 200 minutes | 250 minutes |
| Corrections and exceptions | 100 minutes | 150 minutes |
| Specific oversight of the delegated workflow | 0 minutes | 100 minutes |
| Operating total | 900 minutes | 600 minutes |
The observed operating difference is 900 − 600 = 300 minutes, or 5 hours: 9 versus 6 human minutes per accepted case. Looking only at preparation would have credited the project with 500 minutes, ignoring the increase in review and the other interventions.
Now suppose the team also spends a one-off 180 minutes preparing and checking the pilot. Total human effort for the trial becomes 600 + 180 = 780 minutes. The difference from the baseline in the observed period is therefore 120 minutes, or 2 hours. Show this preparation separately without making it disappear or automatically repeating it in every future period.
Cost per case and unfinished work
Calculate cost per accepted case by dividing the costs attributed to the same scope and period by the number of accepted cases. Explain whether you include setup allocated to the trial, the applicable recurring fee, any contracted variable charges, internal time valued for costing and necessary third-party costs. Do not count an already included item twice, and show the total before dividing it.
Open cases have already consumed resources: show them beside the unit cost. If the periods are not comparable, the figure remains provisional. The service cost guide explains quotation components; here, the question is which costs accompany an actually accepted result. Time available for other work and reduced cash expenditure remain different outcomes.
End the trial with a reasoned decision
Prepare a page showing scope, baseline, results, open cases and the comparison’s limitations. Assign an action to each issue: continue within the same boundary, correct and repeat the assessment, or suspend the affected step. The guide to operational controls helps establish who may authorise a restart or an expansion.
The service description includes testing and observing completed work; the format, accessible data and criteria of any pilot need agreement. A successful sample supports that bounded decision. It does not establish a benefit for every department or every month.