Start with the decision
Decide what the pilot must tell you
An AI tool can produce an impressive answer in a demonstration and still fail to improve the work around it. The input may have been unusually clean, the task carefully selected and the human review left out of the clock. The useful question is whether the whole intervention improves a defined outcome compared with what people would otherwise do, at an acceptable level of risk and cost.
Start by naming the decision. Is the organisation deciding whether to stop, change the workflow, extend a limited test or invest in wider deployment? Who owns that decision, and what evidence would change their mind?
Frame an operational problem
“Staff spend time assembling a first draft of a programme paper from approved documents, and reviewers need to verify its sources, completeness and assumptions” is testable. “We need to use AI” is not.
Write down the chain from activity to result: the system receives specified inputs, a person reviews the output, the changed process may affect time or quality, and an accountable owner decides whether those effects justify the cost and risk. Treat every link as a hypothesis to test.
Baseline first
Record the existing process before changing it
A before-and-after comparison is only useful when “before” is properly described. Record the work people do now, the case mix, the inputs available, the people and systems involved, and the quality checks already in place. Use enough examples to understand normal variation; a single showcase task can conceal it.
Work and inputs
Task types, source quality, complexity, volumes, exceptions and systems used.
Time and effort
Elapsed time, hands-on staff time, handoffs, checking and rework.
Quality
Omissions, unsupported statements, traceability, acceptance and rejection rates.
Cost and control
Tooling, integration, support, assurance, permissions and accountable ownership.
The draft-generation timer alone is not the outcome. A workflow that creates a draft in minutes but takes longer to verify may not improve total delivery time. Likewise, fewer staff hours on one step do not by themselves prove that capacity was released or that financial savings occurred.
Make the comparison credible
Compare representative work, not a showcase
State what the pilot is being compared with: the current manual process, an existing non-AI automation, a different workflow or a comparable set of tasks. Record the comparison clearly enough that someone outside the pilot team can interpret the result.
Where it is feasible and appropriate, an experimental design may help estimate the effect. Other settings may suit a quasi-experimental or theory-based approach, especially when the work is complex or the question is how and for whom the process works. Bring evaluation expertise in early where the stakes, uncertainty or public impact justify it.
Avoid selecting only easy tasks for the AI-assisted group and comparing them with the hardest historical work. Record differences in task complexity, input quality, user experience and other process changes happening at the same time.
Measure the whole intervention
Count human review, quality and risk alongside speed
Set acceptable error boundaries before examining the results. A low-risk internal summary and a recommendation affecting an individual need different levels of assurance.
Agree who is accountable for review, what they must verify, when the system should stop or escalate, and how corrections are recorded. Human judgement can remain essential while the workflow creates value; the evaluation should make the division of work visible.
Preserve validity
Record changes and state what the evidence cannot prove
Record the model or service version, relevant configuration, workflow, training and date for each test period. If the intervention changes materially, say so. An early test of one model and process does not automatically predict later performance at scale or in another team.
If the sample is small or the process changes during the test, describe the result as preliminary learning rather than proof of impact. Separate what was observed from the explanation of why it happened.
Keep the claim within the evidence
“The pilot reduced median preparation time for these 40 comparable cases, while review effort remained within the agreed range” is interpretable. “AI transformed productivity” is not.
Agree the rule in advance
Make stop, adapt or scale a real decision
Before the pilot starts, agree the outcome that needs to improve, the quality and safety conditions that cannot be traded for speed, the minimum evidence needed, and the conditions that trigger a pause or human-only fallback.
The value is insufficient, the risk is unacceptable or the evidence is too poor to justify more investment.
Narrow the scope, improve inputs, change controls or run a better comparison.
Extend within defined limits, while measuring transfer, operational load and new risks.
A useful conclusion may be “stop” or “narrow the use”. Those are productive outcomes when they prevent a weak application from absorbing more time and budget.
Start with the evidence
What decision must your AI pilot support?
Bring one proposed pilot, the process it would change and the outcome you need to test. AI Venture X can help define the baseline, comparison, control boundaries and scale decision.
Book a confidential 30-minute scoping call ↗Sources
Primary guidance used in this guide
- UK Government, Guidance on the Impact Evaluation of AI Interventions. Updated 15 May 2026.
- UK Government, AI Playbook for the UK Government. Guidance on AI delivery, meaningful human control and lifecycle management.
- UK Government, Data and AI Ethics Framework. Guidance on accountability, challenge, safety and impacts.
Sources checked 28 September 2026. This is general practical guidance, not legal, evaluation or assurance advice for a particular organisation. It reports no AI Venture X client result or measured pilot outcome.