Every agent vendor arrives with a number: a benchmark score, a pilot result, a demo that works. Most of these are real. Very few of them tell you whether the system will behave when it meets your patients, your data, and your clinicians under time pressure. This page is a working checklist you can hold against any clinical AI agent before it goes live — ten dimensions to test, each tied to a published benchmark or a named safety framework, with the questions to put to a vendor. It is a companion to how to read an AI validation study, narrowed to the specifics of systems that act. As of July 2026.
Why a benchmark score is a starting line
The evidence that scores mislead is itself measurable. On MedAgentBench — 300 physician-written tasks inside a FHIR-compliant virtual EHR — the best of 12 evaluated models completed 69.67% of tasks 1. That is a capable system that still fails roughly three tasks in ten. AgentClinic shows a second gap: when static exam questions are re-posed as an interactive doctor-patient workup, diagnostic accuracy can fall to below a tenth of the static-exam figure 4. The reasoning the exam rewards is different from the sequential data-gathering real care demands, and the benchmark that flatters a model is the wrong one to trust.
Reliability is a separate axis again. τ-bench reran the same tasks across trials and found that state-of-the-art function-calling agents succeed on fewer than 50% of tasks and are inconsistent even when they can succeed — pass^8, the rate of getting a task right across all of eight attempts, fell below 25% in one domain 3. An agent that passes your demo once can still fail one run in three. So the first move in any evaluation is to stop reading the vendor's number and start testing the system on tasks that resemble yours. The cross-model figures live in our LLM medical benchmark results tracker.
The checklist
Work through these ten dimensions before an agent touches a patient. Each maps to something a trustworthy evaluation should produce, and to the primary source that defines the expectation.
| # | Dimension | The question to ask | Anchor |
|---|---|---|---|
| 1 | Task success | Does it complete realistic tasks from your workflow rather than a curated set? | MedAgentBench 1 |
| 2 | Reliability | Is it consistent across repeated trials, or only good on average? | τ-bench pass^k 3 |
| 3 | Tool-call correctness | When it calls an EHR or order tool, are the arguments right and the action reversible? | MedAgentBench 1 |
| 4 | Robustness to messy input | Does it hold up on interactive, incomplete, adversarial data? | AgentClinic 4 |
| 5 | Calibration and abstention | Does it signal uncertainty and stop, or act confidently when wrong? | NIST GenAI Profile 5 |
| 6 | Safety and harm profile | Which of the named generative-AI risks apply, and are they measured? | NIST GenAI Profile 5 |
| 7 | Security | Is it hardened against prompt injection through data it reads? | OWASP LLM Top 10 6 |
| 8 | Human-oversight design | Can a clinician understand, interpret, override, and stop it? | EU AI Act Art. 14 7 |
| 9 | Post-deployment monitoring | Is there a plan to watch performance and drift after go-live? | FDA lifecycle guidance 9 |
| 10 | Compute and latency budget | What does the agent loop cost in tokens and time at your volume? | Decision-agent benchmark 2 |
1. Task success on realistic tasks
Start where the vendor starts, but change the test set. A score on a public benchmark tells you how a system does on that benchmark; it says little about your triage notes, your order sets, your patient mix. MedAgentBench is useful precisely because its tasks are physician-written and run against a realistic EHR, and even there the ceiling was 69.67% 1. Ask for performance on tasks drawn from your own workflow, scored by your own clinicians, before any other number.
2. Reliability across trials
Average accuracy hides variance. An agentic system plans and acts across steps, and a small probability of error at each step compounds. τ-bench's pass^k metric exists to expose this: it measures success across all of k repeated attempts, and it stays low even for strong agents 3. Ask the vendor to run the same tasks many times and report the worst case, not the mean. A system you cannot trust to repeat itself is a system you cannot supervise efficiently.
3. Tool-call correctness and reversibility
An agent's real risk lives in its tool calls. Retrieving a lab value is low stakes; filing an order is not. Evaluate whether the arguments the agent passes to each tool are correct, whether the action is logged, and — critically — whether it is reversible or requires sign-off before it takes effect. MedAgentBench evaluates task completion inside an EHR environment for exactly this reason 1. The design principle to insist on: consequential tools sit behind human approval, and every call is recorded.
4. Robustness to messy and adversarial input
Benchmarks are clean; clinics are not. AgentClinic's core finding — accuracy collapsing when a static question becomes an interactive workup with missing and ambiguous information 4 — is the shape of most real failures. Test the agent on incomplete histories, contradictory notes, and the interruptions of real use. A system that only performs on tidy inputs has been evaluated on a population it will rarely meet, the agent equivalent of the external validation gap that sinks predictive models.
5. Calibration, uncertainty, and abstention
The safest agents know when to stop. A system that acts with uniform confidence, including when it is wrong, is more dangerous than one that flags doubt and escalates. The NIST Generative AI Profile names confabulation — confidently stated false content — as a distinct risk category to be governed and measured 5. This is the agent-scale form of AI hallucination in clinical contexts. Ask whether the system exposes a calibrated uncertainty signal, whether it can abstain, and what it does when it has too little information to act.
6. Safety and harm profile
Move from "does it work" to "what could it do wrong." The NIST Generative AI Profile catalogs generative-AI risks — including confabulation, information integrity, information security, data privacy, and human-AI configuration — and maps mitigations to the framework's four core functions: govern, map, measure, and manage 5. Use it as a coverage list: for each named risk that applies to your deployment, ask what the vendor measured and what mitigation is in place. A safety case that addresses only accuracy has skipped most of the list.
7. Security against prompt injection
An agent that reads external data — a message, a note, a web result — can be steered by that data. The OWASP Top 10 for LLM Applications ranks prompt injection as the number-one risk, covering both direct manipulation and indirect injection through content the model ingests, and recommends defence-in-depth: least-privilege tooling, input and output filtering, human approval for high-risk actions, and adversarial testing 6. In a clinical agent with tool access, this is a patient-safety control, because a successful injection can turn a reader into an actor. See prompt injection in clinical systems for the mechanism.
8. Human-oversight design
Evaluate the oversight, not the model alone. EU AI Act Article 14 requires high-risk systems to be designed so a person can understand their capacities and limitations, remain aware of automation bias, correctly interpret output, decide to disregard or override it, and intervene or stop the system 7. US FDA excludes decision-support software from device regulation only when the clinician can independently review the basis for its recommendations 8. Ask to see the oversight interface, the escalation path, and the stop control — the full pattern is in designing human oversight for clinical agents.
9. Post-deployment monitoring
An evaluation that ends at go-live is half an evaluation. FDA's 2025 draft guidance on AI-enabled device software functions frames expectations across the total product life cycle, including performance monitoring after deployment 9. Agents drift as models, prompts, tools, and patient populations change. Ask what will be logged, what metrics will be watched, who owns the monitoring, and what triggers a rollback. A vendor without a monitoring plan is proposing to deploy blind.
10. Compute and latency budget
The autonomy has a running cost. A 2026 benchmark of agent systems on clinical decision tasks measured more than ten times the token usage and more than twice the latency of the base model inside them, with the strongest dialog configuration reaching 60.3% accuracy 2. At scale, that cost and delay shape whether the agent is usable at the bedside. Ask for the token and latency profile at your expected volume, because an agent that is accurate but too slow or too expensive to run consistently will be switched off.
A pre-deployment evidence table
Turn the checklist into a request. Ask a vendor to fill this in, with a source or a test protocol for each cell — the empty cells are as informative as the full ones.
| Dimension | Evidence to request | Red flag |
|---|---|---|
| Task success | Accuracy on your-workflow tasks, scored by your clinicians | Only public-benchmark numbers |
| Reliability | Worst-case success across repeated trials | Mean-only reporting |
| Tool calls | Log of tool calls with argument-level correctness | No action logging |
| Robustness | Performance on incomplete and adversarial inputs | Sandbox-only results |
| Calibration | Uncertainty signal and abstention behaviour | Uniform confidence |
| Safety | NIST-risk coverage with measured mitigations | Accuracy-only safety case |
| Security | Prompt-injection test results | No adversarial testing |
| Oversight | Oversight interface, escalation, stop control | Oversight asserted, not shown |
| Monitoring | Post-deployment metrics and rollback trigger | No monitoring plan |
| Cost | Token and latency profile at your volume | Unstated compute cost |
How to read this checklist
Three cautions travel with it. First, no single dimension is disqualifying on its own — a modest task-success number under strong oversight can be safer than a high one with none. Weigh the answers together. Second, the checklist assesses the evidence, not the demo: a well-tested agent that performs modestly is more trustworthy than a dazzling one that has only ever run in a sandbox — the same logic that governs any validation study. Third, the anchors evolve. The framework and benchmark names here are current as of July 2026, and the underlying questions — does it work, does it repeat, can we catch it when it fails — outlast any one revision.
Sources and method
This checklist maps each test to a primary source: four agent benchmarks that bound capability and reliability 1234, the NIST Generative AI Profile for the risk taxonomy 5, the OWASP LLM Top 10 for security 6, EU AI Act Article 14 and FDA guidance for oversight 78, FDA lifecycle guidance for monitoring 9, and the automation-bias literature that explains why oversight must be designed rather than assumed 10. Every figure is drawn from the primary source cited beside it. We revisit this page on a 180-day cycle and whenever a benchmark or framework named here is materially revised. For the concept, see what is agentic AI in healthcare; for the oversight half of deployment, see designing human oversight for clinical agents.