An AI agent is production-ready when it passes ten tests: correct task outcomes, safe tool actions, injection resistance, enforced permissions, grounded answers, acceptable latency, failure recovery, stable multi-turn state, human escalation, and traceable regression coverage.
Each test below states what to run, what passes, and what should block launch. Use it as a release gate for AI engineering projects, whatever model or framework you use.
Why agent testing differs from prompt testing
Agents call tools, change records, and adapt across many turns, so one early mistake can compound. Anthropic's engineering guide warns that teams without evals can get stuck in reactive loops, catching issues only in production.
Grade the outcome, not only the reply. A booking agent can announce a confirmation while the reservation system holds nothing.
Repeat every scenario. In Anthropic's example, an agent with a 75% per-trial success rate passes three trials in a row only about 42% of the time. Consistency is the real bar for customer-facing agents.
Start small. Anthropic suggests 20 to 50 tasks drawn from real failures, then grow the suite as incidents appear.
The 10 pre-production tests
1. Functional accuracy and task completion
Run representative tasks, edge cases, ambiguous or incomplete requests, and requests the agent should decline. Define the expected outcome for each case before testing, then score against a written rubric and the final system state.
Pass when: critical workflows succeed across repeated trials, and every failure is labeled by the step where it broke.
2. Tool and API action correctness
Verify the agent picks the right tool, sends valid parameters, and handles responses correctly. Replay timeouts and duplicate requests against your integrated systems, using a sandbox with synthetic records, never live production data.
Pass when: one logical request creates one record, and malformed or wrong-tool calls are rejected.
3. Prompt injection and data leakage
Test direct attacks such as “ignore previous instructions”, and indirect injection hidden in documents, emails, web pages, and tool outputs. Also try to extract system prompts and secrets.
OWASP says RAG and fine-tuning do not fully mitigate prompt injection, and recommends adversarial testing that treats the model as an untrusted user.
Pass when: injected instructions never trigger unauthorized retrieval, disclosure, or actions.
4. Permissions and least privilege
OWASP's excessive agency risk traces to excessive functionality, permissions, or autonomy. Call every write, delete, export, and send endpoint with the agent's normal credential, across roles and tenants. Include cross-tenant reads, expired sessions, and revoked permissions.
Pass when: the backend denies anything outside the grant, whatever the model requests, and high-impact actions require approval.
See LuMay's guide to agent access controls and its security architecture.
5. Hallucination, grounding, and retrieval quality
NIST's Generative AI Profile calls confidently stated false content confabulation, including invented citations. Ask answerable and unanswerable questions, plus conflicting and outdated sources, and check each claim against retrieved sources.
For RAG, test retrieval before generation: did the right, permitted, current passages arrive? A polished answer cannot make up for retrieving the wrong document. See RAG development and the LLM evaluation metrics guide.
Pass when: claims trace to permitted sources, and the agent abstains when evidence is missing.
6. Latency and load
Time each request from user input to completed action, not first token. Report p50, p95, and p99 with sample counts at expected peak concurrency, and include retrieval, tool calls, retries, guardrails, and approval pauses.
Pass when: percentiles meet targets set from your baseline and users' tolerance, and error rates hold under load.
Voice agents add interruption tests. See voice agent testing and LuMay Voice Agents.
7. Failure recovery
Break dependencies on purpose: timeouts, server errors, empty search results, expired tokens, and a success response that never arrives.
Pass when: the agent retries within limits, avoids loops and duplicate actions, reconciles uncertain outcomes, and tells the user truthfully what happened. It must never claim an action succeeded unless a trusted system confirms it.
LuMay's agent development page lists step limits and fallback paths as controls to verify.
8. Multi-turn and state handling
Script long conversations with mid-task corrections, topic switches, resumed sessions, and contradictory instructions. Anthropic notes that conversational agent evals often use a second model to simulate the user.
Pass when: the latest correction wins, approved facts persist, the agent re-checks the system of record after changes, and state never leaks between users or sessions.
9. Human escalation and handoff
Trigger every escalation path: user request, low confidence, repeated failure, policy boundary, and high-impact actions. For each one, define the trigger, the destination queue or role, the context handed over, and the message the user sees. Confirm a person actually accepts the handoff.
Pass when: the summary contains authorized fields only and no user is stranded.
LuMay's Legal Agents page describes attorney approval over generated actions, a pattern worth testing in any regulated workflow.
10. Observability and regression readiness
Pick a finished run and reconstruct it from traces alone: request, prompt and model version, retrieved sources, tool calls, approvals, and outcome. OpenTelemetry's GenAI conventions define spans, metrics, and events for GenAI systems, so confirm which version your stack emits.
Then turn every failure into a permanent regression test. Anthropic says regression evals should have a nearly 100% pass rate.
Pass when: a reviewer can explain any run without the developer, and the suite reruns after every prompt, model, tool, retrieval, or policy change.
See audit trails and LLMOps practices.
Choose the right way to grade each test
| Method | Best for |
|---|---|
| Deterministic check | Tool arguments, final system state, schema validity, latency, permissions |
| LLM judge | Tone, groundedness, and open-ended quality, calibrated against human review |
| Human review | Ambiguous edge cases, high-risk actions, and judge calibration |
Use code to verify anything code can verify. Reserve model judges for what it cannot.
Turn the checklist into a release gate
| Severity | Example failure | Launch rule |
|---|---|---|
| Blocker | Unauthorized action, data exposure, or duplicate irreversible action | No launch until fixed and retested |
| Major | Failed handoff strands a user, or recovery fails on a key workflow | Fix, or narrow scope with human review |
| Minor | Awkward wording or small latency misses | Ship with an owner and a date |
Set thresholds from your own risk and baseline. No universal passing score exists. Before launch, also name a rollback decision owner, monitoring thresholds, and an incident runbook.
Conclusion: ship agents you can prove
Production readiness is evidence, not confidence. Run all ten tests, fix blockers, and keep the suite running after launch.
LuMay's AI engineering services and AI agent development cover testing of task completion, tool accuracy, edge cases, safety, and prompt injection scenarios. Book an AI engineering consultation to discuss your agent's release criteria.
