An AI agent rarely fails with a crash. It fails quietly. It sends an email it should have drafted, calls a tool twice, quotes a policy that does not exist, or burns through a month of model spend overnight.
Ordinary incident runbooks assume a service is up or down. Agents need a runbook that asks a different question: what authority did the agent use, and how do we take that authority away without breaking everything else?
This guide gives engineering teams that runbook. It covers what counts as an agent incident, a severity matrix, a first-hour checklist, containment options, the evidence to keep, safe recovery and a postmortem template you can copy.
What Counts as an AI Agent Incident
Treat any agent acting outside its stated purpose as a possible incident until you know otherwise. Filing it as a bug ticket delays containment and loses evidence.
• Unsafe action: the agent wrote, sent, paid, deleted or changed something it should not have
• Data exposure: the agent revealed or moved data to a person or system without the right access
• Hijack: instructions inside a document, email or web page changed the agent's behavior (prompt injection)
• Wrong answer at scale: a confident, repeated error reaching customers or staff
• Runaway cost or loops: tool calls or tokens far above the normal pattern
• Control failure: an approval step, kill switch or audit log did not work as designed
The last category matters most. A single wrong answer is a quality problem. A broken control means the next failure will also go unnoticed.
Before Anything Breaks: Five Things to Have Ready
A runbook is only as fast as the switches it can reach. Put these in place during build, not during the incident.
➤ A per-agent kill switch that stops new runs and cancels queued tool calls
➤ Per-tool toggles so you can disable "send_email" without disabling "search_kb"
➤ A read-only mode that keeps the agent answering but blocks every write action
➤ Version pinning for the model, prompt and tool definitions, with one-click rollback
➤ Traces with IDs linking each user request to retrieved context, tool calls and outputs
If you cannot answer "which version was serving at 14: 05?" within two minutes, fix that first. Every later step depends on it.
For prevention controls such as least-privilege tools and input filtering, see our guide on protecting agents from prompt injection and unsafe tool actions. This runbook starts where prevention ends.
Severity Matrix for Agent Incidents
Severity drives who gets paged and how fast you contain. The levels below are a starting point; set your own response targets with your on-call and legal teams.
| Severity | Typical trigger | First containment move | Who is involved |
|---|---|---|---|
| SEV1 | Unauthorized external action or confirmed sensitive data exposure | Kill switch, revoke credentials | Incident lead, security, legal, agent owner |
| SEV2 | Control bypass (injection succeeded, approval skipped) with no confirmed harm yet | Disable affected tool or force read-only | Incident lead, security, agent owner |
| SEV3 | Repeated wrong answers or abnormal cost, no unsafe action | Roll back prompt or model version | Agent owner, on-call engineer |
| SEV4 | Single odd output, logging gap, config drift | Ticket with trace ID | Agent owner |
Microsoft's open-source Agent Governance Toolkit publishes a similar four-level scheme with an incident category list. It is a useful cross-check when you write your own.
The First Hour: A Step-by-Step Checklist
Minutes 0–10: Declare and Name a Lead
• Declare an incident in one channel and name a single incident lead
• Start a written timeline using one clock (UTC is simplest)
• Record the agent name, owner, stated purpose and the first odd action with its trace ID
• Announce a change freeze: no prompt, model or config edits except containment
The freeze matters more for agents than for normal services. A well-meant prompt tweak during the incident destroys your ability to reproduce what happened.
Minutes 10–30: Contain the Smallest Responsible Capability
Shutting the whole agent down is safe but costly. Containing the specific capability is usually enough. Work down this ladder and stop at the first step that removes the risk.
- Disable the single tool involved (for example, outbound email)
- Switch the agent to read-only or "draft for approval" mode
- Roll back to the last known-good prompt, model and tool version
- Revoke or narrow the credential the agent used
- Hit the kill switch for the agent
- Block the upstream source (a poisoned document, inbox or URL) from retrieval
Then confirm containment with live behavior, not a settings screen. Send a safe test request and check that the blocked action no longer appears in traces.
Minutes 30–60: Scope and Notify
• Query traces for the same tool, user, document or pattern across the full exposure window
• List every external side effect: messages sent, records changed, payments made
• Decide which side effects can be reversed and assign an owner to each
• Brief legal and privacy if personal or client data may be involved; they decide on notification duties
Reporting clocks in many regimes start when you become aware of an incident. Keep the awareness time in the timeline and let counsel interpret it.
Evidence to Preserve
Agent incidents are reconstructed from traces. Freeze log rotation for the affected agent and copy these artifacts to restricted storage.
| Artifact | Why it matters |
|---|---|
| Full trace for each affected session (inputs, retrieved chunks, tool calls, outputs) | Shows the exact path from input to action |
| Prompt, model and tool definition versions in service | Lets you reproduce the behavior later |
| Retrieved documents and their source IDs | Finds poisoned or wrong source content |
| Approval records and policy decisions | Shows whether a control fired or was skipped |
| Credential and permission state at the time | Shows what the agent was able to do |
| Cost and rate metrics around the window | Separates loops from normal load |
Minimize copies of personal data while you do this. Restrict access to the incident team and record who viewed what.
Diagnose the Layer That Failed
Most agent incidents trace back to one of six layers. Naming the layer stops teams from "fixing the prompt" when the real fault was a permission.
• Input: a malicious or malformed instruction entered through user text, a file or a web page
• Retrieval: the wrong, stale or poisoned chunk was fed to the model
• Model: a provider update or sampling change altered behavior
• Prompt: instructions were ambiguous, contradictory or missing a "never" rule
• Tool: the tool allowed more than intended, lacked validation or was not idempotent
• Control: approval, rate limit, budget cap or logging did not work
Reproduce the failure in a staging copy with the preserved versions before you change anything. If you cannot reproduce it, say so in the record rather than guessing.
Recover in Stages
Restore authority one step at a time and watch real traffic between steps.
- Ship one bounded fix aimed at the diagnosed layer, described in a single sentence
- Rerun your regression suite plus the new incident scenario
- Re-enable tools in read-only or approval mode first
- Watch traces and cost for an agreed period
- Restore full autonomy only when the incident lead and agent owner both sign off
Do not close the incident because the fix deployed. Close it when live behavior shows the problem has stopped.
Worked Example: The Refund Agent That Over-Credited
This is an illustrative scenario, not a real customer case. A support agent can issue account credits up to a fixed limit without approval. On a Monday morning, finance notices credits are running well above the usual daily pattern.
• Declare (0–10 min): on-call engineer becomes incident lead, opens a channel and freezes changes
• Contain (10–20 min): the "issue_credit" tool is disabled; the agent keeps answering questions and drafts credit requests for human approval
• Scope (20–50 min): traces show credits were issued twice on some tickets after a timeout and retry
• Diagnose: tool layer; the credit API call was not idempotent, so retries created duplicates
• Fix: add an idempotency key per ticket and a daily credit cap at the tool level
• Recover: tool re-enabled in approval mode for two days, then restored with the cap
Notice what did not change: the prompt. The model behaved as instructed. The failure was in how the tool handled retries, which is why naming the layer matters.
Copy-Ready Incident Record Template
• Incident ID / severity / status:
• Incident lead / agent owner / scribe:
• Agent name, purpose and versions serving (model, prompt, tools):
• Detected by and detection time (UTC):
• Awareness time for reporting purposes (UTC):
• What the agent did, in one sentence:
• Containment steps taken, with times:
• External side effects and reversal owner for each:
• Failed layer (input, retrieval, model, prompt, tool, control):
• Fix shipped and version:
• Verification evidence on live traffic:
• Regression test added (ID):
• Follow-up actions, owners and due dates:
Postmortem Questions That Prevent Repeats
Run a blameless review within a week. These questions focus on controls rather than people.
○ Which control should have stopped this, and why did it not?
○ How long from first odd action to detection, and what signal would have cut that time?
○ Could we contain the capability without stopping the whole agent?
○ Is the scenario now in the regression suite that runs before every prompt or model change?
○ Do other agents share the same tool, credential or data source?
Our guide to LLM observability metrics covers the monitoring signals that shorten detection time.
Map the Runbook to Recognized Frameworks
You do not need a new framework for agents. Extend the ones your security team already uses.
• NIST SP 800-61 Rev. 3 gives incident response recommendations aligned to the NIST Cybersecurity Framework 2.0
• The NIST AI Risk Management Framework covers managing AI risks across the lifecycle
• The OWASP Top 10 for LLM Applications names common failure types such as prompt injection and excessive agency
• Google's SRE chapter on managing incidents describes the incident-lead role and a living incident document
Common Mistakes
○ Treating the first odd action as a bug ticket instead of an incident
○ Editing the prompt before preserving traces and versions
○ Killing the whole agent when one tool toggle would do
○ Trusting a settings screen instead of testing live behavior
○ Closing the incident on deploy rather than on verified behavior
○ Leaving the scenario out of the regression suite
Our Own Product Note
Disclosure: this is our own service. LuMay's AI agent development work includes monitoring and ongoing incident investigation after launch. The runbook above works with any agent stack, whoever built it.
Start With the Switches
Before your next agent launch, confirm you have a kill switch, per-tool toggles, read-only mode, version pinning and traced sessions. Those five items decide how fast the first hour goes. For help designing them, see our AI engineering services.
References
• NIST SP 800-61 Rev. 3, Incident Response Recommendations: https://csrc.nist.gov/pubs/sp/800/61/r3/final
• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
• OWASP Top 10 for LLM Applications: https://genai.owasp.org/llm-top-10/
• Google SRE Book, Managing Incidents: https://sre.google/sre-book/managing-incidents/
• Microsoft Agent Governance Toolkit, Incident Response Workflow: https://microsoft.github.io/agent-governance-toolkit/compliance/incident-response-workflow/
