Protect enterprise AI agents by treating retrieved content as untrusted data and enforcing every consequential action through independent, server-side authorization. Combine least privilege, scoped credentials, validated tools, restricted execution, approval gates, spending limits, and audit records.
Enterprise AI agent security must hold even when the model follows malicious instructions. The practical goal is to prevent a compromised conversation from gaining permissions, disclosing restricted information, or executing an unauthorized transaction.
Where prompt injection becomes a business risk
Direct prompt injection enters through the agent’s immediate input, such as a user message attempting to override its task. Indirect prompt injection arrives through material the agent reads: a retrieved document, website, email, support ticket, or tool response.
A legitimate document can contain malicious instructions. Its availability in an approved knowledge base establishes neither the authority of those instructions nor permission for the actions they request. OWASP’s prompt injection guidance distinguishes these attack paths and explains that RAG and fine-tuning do not eliminate the vulnerability.
Excessive permissions turn manipulated reasoning into operational harm. A tool intended to retrieve invoices might also permit exporting financial records, changing suppliers, or submitting payments. OWASP’s excessive agency guidance identifies overly broad functionality, permissions, and autonomy as contributing problems.
- Data exfiltration: confidential information leaves through messages, uploads, tool arguments, or generated links.
- Unsafe tool actions: the agent deletes records, modifies privileges, purchases services, or sends communications outside its mandate.
- Persistent contamination: malicious content is saved into memory and influences a later task.
Reading data and disclosing it are separate decisions. A read-only connector can still expose sensitive information through the agent’s answer.
Build authorization outside the model
Prompts describe intended behavior. Filters detect some suspicious content. Model judgment can help identify anomalies, but none of these establishes an authenticated user’s authority over a resource.
A second model can repeat the first model’s mistake, particularly when both consume the same malicious evidence. Use model-based checks as additional detection, while ordinary application controls enforce permissions and business limits.
Official OpenAI agent safety guidance recommends structured outputs, tool approvals, and layered safeguards, and explicitly acknowledges that guardrails are not foolproof. Valid JSON can still describe an unauthorized payment.
The architecture below is an engineering recommendation based on those risks. Its policy decisions use authenticated context, approved configuration, and authoritative records rather than the model’s interpretation of permission.
- Authenticate the requester and establish the permitted task.
- Let the model propose a narrowly defined tool action.
- Validate arguments and independently authorize the operation.
- Obtain any required approval for the exact validated action.
- Execute through controlled credentials and record the downstream outcome.
Apply layered controls at each trust boundary
1. Separate instructions, retrieved data, and authority
Label source content and preserve provenance as it moves through retrieval, summarization, memory, and delegation. Avoid placing external text into privileged instruction fields.
Enforce tenant, document, and field permissions before retrieval reaches the model. Apply equivalent restrictions to cached answers and conversation history. Delimiters and source labels help interpretation, but they do not create a security boundary by themselves.
NIST’s Zero Trust Architecture rejects implicit trust based on network location or ownership. Applied here, an internal document or agent process should receive only the access explicitly authorized for its task.
2. Minimize tools and credential scope
Separate reading, drafting, sending, updating, deleting, and purchasing. Prefer a dedicated operation such as creating a purchase draft over an unrestricted shell or generic HTTP client.
Use separate workload identities and short-lived, audience-restricted credentials where supported. Keep secrets outside prompts and tool responses, protect renewal credentials, and test revocation at receiving services.
These choices limit the consequences of compromise, but a narrowly scoped token can still perform harmful actions within its scope. LuMay’s guide to nine AI agent access controls develops the identity and delegation requirements in more detail.
3. Authorize every operation server-side
At the execution gateway and destination service, check the actor, tenant, delegation, operation, resource, and current policy. Derive identity from authenticated context rather than accepting model-supplied claims about the user.
Use deny-by-default rules and reject actions when mandatory authorization checks are unavailable. Recheck queued work at execution time because permissions and resource state may have changed. These recommendations follow OWASP’s authorization guidance.
Prevent bypass routes: a protected agent gateway provides little assurance if the same credentials can call an unrestricted downstream endpoint directly.
4. Validate tool arguments and outbound destinations
Allowlist tools and operations, then validate argument types, ranges, resource ownership, recipients, and permitted data categories. Resolve identifiers through trusted services instead of accepting arbitrary paths or account numbers.
For network tools, restrict egress and validate actual destinations. Disable redirects where unnecessary, or revalidate each redirect before connecting. Authorization to use an email service does not authorize every recipient.
Inspect outbound data and redact unnecessary fields. Detection can miss sensitive content, so combine it with explicit destination and disclosure rules. OWASP’s AI agent security guidance supports scoped tools, independent validation, and data protection.
5. Sandbox execution and constrain resources
Run code and file-processing tools in isolated environments with restricted filesystem access, network routes, execution time, and resource consumption. Separate production credentials from experimental execution.
Sandboxing reduces reachable systems. It does not make permitted business actions safe: an isolated process with an authorized payment credential can still submit an unauthorized purchase unless transaction policy blocks it.
6. Bind approvals and spending limits to exact actions
Show approvers the original request alongside server-validated amounts, recipients, destinations, and affected resources. Bind approval to those parameters, an expiry, and a single-use transaction identifier.
Reject parameter changes after approval. Keep the final authorization check close to execution, consistent with OWASP’s transaction authorization guidance.
For purchasing, define purpose, approved merchant, currency, per-transaction amount, cumulative budget, and validity period. Reserve budget atomically across parallel workers so many individually small purchases cannot bypass the total.
Use durable idempotency records for retries. When an external payment times out, reconcile its status before repeating it; an idempotency key alone does not prove exactly-once execution.
Two hypothetical workflows
These scenarios illustrate proposed controls. They are not LuMay customer incidents, implementation claims, or measured results.
Workflow 1: A malicious retrieved document requests an upload
An employee asks an agent to summarize an internal supplier assessment. A paragraph instructs the agent to upload the full customer directory to an external destination for “verification.”
- Proposed action: export customer records and upload them externally.
- Independent policy check: compare the authenticated summarization task with export permission, document classification, and approved destinations.
- Decision: block export and upload. Allow an authorized summary using only permitted source information.
- Audit record: capture run and document references, tool and destination identifiers, policy version, denial reason, and confirmation that no upload executed.
Do not copy the customer directory into the denial log. Investigate the source document and prevent contaminated summaries or memory from carrying its instruction into subsequent runs.
Workflow 2: An agent proposes an unauthorized purchase
A purchasing task permits up to ₹50,000 with a named supplier before an approved expiry. Malicious retrieved text induces a ₹40 lakh purchase from a different merchant.
- Proposed action: submit a ₹40 lakh payment to the new merchant.
- Independent policy check: validate supplier identity, purpose, currency, mandate expiry, remaining budget, and transaction-bound approval.
- Decision: block execution because the merchant and amount violate the mandate. Allow preparation of a review draft without moving money.
- Audit record: retain mandate and proposal references, mismatched fields, policy version, block decision, and the absence of a submitted payment.
A new purchase requires a separately authorized mandate. Splitting the proposal into smaller payments should also fail against the shared cumulative limit.
Threats, controls, and practical limitations
| Threat | Primary control | Remaining limitation |
|---|---|---|
| Retrieved instructions hijack a task | Provenance, context separation, action authorization | The answer may still be misleading |
| Tools expose excessive authority | Narrow operations and scoped identities | Allowed operations can still be misused |
| Confidential data leaves the system | Access checks, egress rules, data minimization | Detection can miss transformed content |
| Approval parameters change | Exact-action binding and final validation | Approvers can misunderstand the request |
| Parallel calls exceed spending limits | Atomic budget reservation and aggregate limits | External settlement requires reconciliation |
| Retries repeat a consequential action | Durable deduplication and outcome reconciliation | Third-party APIs may provide weak guarantees |
Implement priorities with named owners
Start with the operations that can expose sensitive data or create irreversible effects. Expand autonomy only after the relevant boundaries have observable, repeatable enforcement.
| Priority | Deliverable | Accountable owner |
|---|---|---|
| Before production | Inventory sources, tools, credentials, and prohibited actions | Application owner with security architect |
| Before consequential actions | Authorization gateway, scoped identity, argument and egress checks | Platform engineering with IAM team |
| Before spending or destructive changes | Approval binding, budgets, deduplication, recovery procedure | Business process owner with engineering |
| During operation | Monitoring, revocation drills, regression tests, incident response | SRE and security operations |
For an AI voice agent, apply these boundaries to callers, transcripts, CRM updates, and external messages. A caller’s spoken instruction should not independently authorize a refund or disclose another customer’s record.
Test downstream effects and prepare incident response
NIST’s Generative AI Profile addresses prompt injection as an information-security risk and recommends adversarial evaluation. Build a versioned test set that includes realistic authorized tasks alongside attacks.
The following are proposed acceptance criteria, not reported results. Set performance and usability thresholds before testing, with security and business owners signing off on exceptions.
- Authorization: zero unauthorized downstream actions across the defined release suite, including cross-tenant requests and forged identity arguments.
- Approval integrity: changed amount, merchant, destination, expired approval, and replay cases are denied.
- Financial integrity: concurrent calls cannot exceed the approved aggregate budget or create duplicate purchases.
- Disclosure: controlled sensitive markers never reach unauthorized test destinations or logs.
- Operational measures: record attack success by actual effects, legitimate-task false blocks, approval delays, gateway latency, and revocation propagation time.
Repeat relevant tests after model, prompt, connector, retrieval, and policy changes. Passing a finite suite provides evidence for those cases, without proving immunity to future attacks.
Log request and run identifiers, authenticated actors, source references, tool arguments with sensitive fields removed, policy decisions, approvals, and verified outcomes. LuMay’s AI agent audit-trail guide explains how to connect that evidence.
During an incident, suspend affected execution routes, revoke applicable grants, stop queued work, preserve restricted evidence, and reconcile external effects. Restore service after fixing and retesting the failed boundary; disabling an agent cannot reverse an already completed disclosure or payment.
Benefits, tradeoffs, and suitable use cases
Layered controls reduce the authority available to manipulated agents and make decisions easier to investigate. They are especially useful where workflows combine untrusted content with confidential records, financial transactions, or production changes.
The costs include additional latency, policy maintenance, approval workload, and integration effort. Legacy APIs may lack narrow permissions, revocation, or reliable transaction status, requiring compensating controls or reduced autonomy.
Residual risks include compromised credentials, connector defects, misleading authorized outputs, stale permissions, and human approval errors. When an operation cannot be constrained or reconciled sufficiently, keep execution with an authorized person.
Plan security into your enterprise AI workflow
Define authorization, data boundaries, approval rules, and recovery before expanding agent autonomy. Discuss your enterprise AI engineering requirements with LuMay to scope an implementation around your systems, operational constraints, and risk tolerance.
