A CRM administrator changes a required field. The integration still returns HTTP 200, but new leads arrive without the routing value that assigns an owner.
The agent tells the user the lead was created successfully. Sales never sees it.
AI tool contract testing checks the agreement between an agent-facing tool and the business service behind it. It should cover accepted arguments, authorization, business outcomes, and the meaning of responses—not merely whether JSON parses.
Write down the contract at both boundaries
An agent sees a tool name, description, and input schema. The adapter sees the actual downstream API. Both boundaries can change independently.
For each tool, document required fields, allowed values, identity sources, side effects, failure states, and the evidence that establishes success. Map every field the adapter transforms.
For a lead-creation tool, a useful success contract might require a CRM record ID and confirmed routing status. If assignment is asynchronous, return that state honestly instead of representing the whole process as complete.
Validate shape before business meaning
JSON Schema's object reference explains that declaring a property does not make it required. It also documents how additional properties are handled. Review these behaviors explicitly when defining tool inputs.
Schema validation can reject an unknown priority value or a missing identifier. It cannot establish that the authenticated caller may update that record or that the requested transition is appropriate.
| Test layer | Question |
|---|---|
| Schema | Are the fields, types, and values accepted? |
| Authorization | May this identity act on this tenant and record? |
| Business rule | Is this change permitted in the current workflow state? |
| Adapter mapping | Are fields translated without losing meaning? |
| Outcome | Did the intended business operation actually occur? |
Keep these checks outside the model's judgment. The model can propose arguments; the service decides whether to execute them.
Create fixtures for valid and invalid requests
Begin with one representative valid request, then vary it deliberately. Remove required fields, send empty strings, use unexpected enum values, supply an identifier from another tenant, and request an invalid state transition.
Also test values that look plausible but mean something different: local time without a timezone, money without a currency, or a customer-facing label where an internal ID is required.
For each fixture, specify the expected outcome and whether any write is allowed. A rejected request must not leave a partially created record behind without a documented recovery state.
Test responses as carefully as inputs
An adapter can fail by interpreting a response incorrectly. Include a nominal success response with a missing business ID, a partial batch failure, a validation error, a timeout after submission, and a provider response containing an unfamiliar optional field.
Unknown response fields should not automatically break every integration. Missing critical evidence should. Decide which fields are essential to your contract and tolerate additive changes only where the semantics remain clear.
Use sanitized fixtures and a sandbox service where possible. Do not run write tests against live customer accounts just to make the test resemble production.
Illustrative example: an assignment rule changes
A hypothetical distributor uses an agent to create service leads. The CRM adds a required region code, but the agent-facing schema still treats location as free text.
A contract test submits the old fixture. It verifies both record creation and assignment to the expected queue. The assignment check fails, exposing the mismatch before the updated integration is released.
The fix maps a validated location to the current region identifier and returns an explicit error when that mapping is unavailable. It does not invent a default region to make the request pass.
Version the meaning, not just the JSON
A field can keep its name while changing its meaning. A status called “closed” might now refer to a resolved case rather than an abandoned request.
Have business owners review semantic changes. Maintain supported schema versions where clients cannot upgrade together, and make deprecation visible. Keep descriptions, runtime validation, fixtures, and adapter mappings aligned through the release process.
For broader integration requirements, see LuMay's guide to connecting AI with legacy enterprise systems.
Frequently asked questions
Are mocked API responses enough for tool contract testing?
They are useful for repeatable adapter checks, but they only test the behavior represented in the fixtures. Complement them with authorized sandbox checks against the real service, including permissions and business outcomes. Refresh fixtures when the provider or workflow changes.
Can adding an optional field still break an integration?
Yes, if a strict consumer rejects unknown fields or the new field changes how the business outcome must be interpreted. Test additive changes against the actual consumer. Decide deliberately which unknown fields are tolerated and which missing facts prevent a success result.
How do you test an HTTP 200 response that represents an incomplete task?
Assert the business evidence, not only the status code. For example, check the record ID and assignment state required by the contract. If assignment is pending, the tool should report pending rather than tell the user the entire workflow is complete.
When should a contract failure block a release?
Block release when it exposes an unauthorized write, lost required information, an invalid business transition, or a misleading success result. Use business owners to assess semantic differences. A cosmetic change to an optional field may need a different response from a broken authorization boundary.
Make contract failures actionable
Record the tool version, adapter version, sanitized failure reason, and correlation ID. Distinguish a bad request from unavailable infrastructure and an uncertain write outcome.
Start with the three tools whose mistakes have the highest operational cost. Define what each success response proves, then test the cases where that proof is missing. That is a stronger release gate than asking whether the agent's final sentence sounds correct.
