Your agent’s new prompt produces clearer answers. The demo looks better. Then a workflow that used to ask for approval starts making the update immediately.
AI agent regression testing checks whether changes to prompts, models, tools, retrieval or orchestration break behavior that previously worked. The practical unit of comparison is a business scenario: the same starting conditions, a defined permitted outcome, and evidence from both the baseline and candidate versions.
This guide proposes a framework-independent method for comparing releases. It focuses on version-to-version behavior; use LuMay’s production testing checklist for broader readiness coverage.
Define the change before running the suite
Record what changed and what should remain stable. A prompt edit, model upgrade and tool-schema update made together can make a failure difficult to explain. Where practical, compare one deliberate change at a time. If a release contains several changes, preserve the complete configuration and investigate failed cases against that manifest.
A release manifest should identify the application revision, model identifier and available version information, prompt version, tool and adapter versions, retrieval configuration, dataset version and evaluator version. Record provider settings that affect generation. If the provider does not expose an immutable model snapshot, record that limitation beside the comparison.
Build scenarios around outcomes and boundaries
Start with workflows users already rely on and failures the team has observed. For each scenario, specify both the expected result and what must not happen. A customer-service agent may need to prepare a change for approval while leaving the customer record untouched.
Comet’s introduction to Opik Test Suites describes concrete assertions and adding failure traces to a growing suite. The useful principle is to turn a discovered failure into a repeatable check with an identifiable broken rule. The method below does not depend on using Opik.
| Scenario field | What to record |
|---|---|
| Case ID and purpose | Stable reference and the behavior being protected |
| Starting state | Test records, permissions, prior conversation and relevant clock conditions |
| Input and permitted outcome | User request and the expected business state |
| Prohibited effects | Writes, disclosures or skipped approvals that must not occur |
| Checks | Backend assertions, response rubric and any human review |
| Baseline and candidate | Version references and results for each repeated trial |
| Decision evidence | Failed rule, trace reference, reviewer and disposition |
Keep ordinary requests, missing information, dependency failures and attempts to cross an authorization boundary in the suite. Group cases by workflow and business risk so an improvement in routine answers cannot conceal a failure in a restricted action.
Reset the environment before comparing versions
A comparison is hard to interpret if the baseline changes the data the candidate receives. Synthesized’s regression testing guide explains restoring comparable starting data before rerunning agent scenarios and comparing end states.
Use isolated test records or a documented reset procedure. Reset relevant business state, conversation state and retrieval inputs before each trial. Freeze time-dependent inputs where practical. Keep actions inside authorized test environments, with integrations configured for test accounts.
Identical starting data does not make a stochastic agent deterministic. Repeat trials and record external dependencies that cannot be held constant. Otherwise, an observed difference may reflect test conditions or run-to-run variability rather than the release change.
Separate hard assertions from judgment
Use application checks for facts the application can establish: which record changed, whether approval exists, whether a forbidden write occurred, and whether the agent returned a confirmed downstream reference. Use a reviewed rubric for qualities such as explanation clarity or whether an answer is adequately supported.
An LLM judge can help assess open-ended responses, but its grade is another measurement with limitations. Compare it with human labels on representative cases, preserve its configuration and inspect disagreements. A favorable language score should not overrule a failed authorization assertion.
Run a paired comparison with a recorded decision
- Validate the baseline. Run the accepted release on the selected cases and confirm the harness and fixtures work. Investigate unexpected baseline failures before interpreting candidate results.
- Run the candidate from equivalent starting states. Use the same scenarios, checks and repeat count; retain each trial rather than only a combined average.
- Compare by case and failure type. Identify newly failing assertions, unstable outcomes, improvements and behavior that changed without an approved requirement change.
- Apply the release policy. Block the candidate on defined critical failures. Review acceptable quality tradeoffs separately, with an accountable approver.
- Retest the fix on held-back cases as well as the known failure. Preserve the original failing scenario so the repair does not erase the regression history.
Choose repeat counts and tolerances according to risk, variability and evaluation cost. There is no universal number of trials that proves reliability. Keep the number attempted, number passing and failure categories visible; a small sample can leave material uncertainty.
Worked example: the better answer bypasses approval
Hypothetical example: a support agent can prepare a service-plan change, but a supervisor must approve execution. A prompt revision encourages the agent to finish tasks with fewer conversational turns.
| Scenario | Baseline result | Candidate result | Proposed decision |
|---|---|---|---|
| Eligible change with approval | Approved record updated | Approved record updated | No regression observed in these trials |
| Eligible change without approval | Request queued; record unchanged | Record updated in one trial | Block release and investigate the bypass |
| Unknown customer identifier | Clarification requested; no write | Clarification requested; no write | No regression observed in these trials |
| Update service unavailable | Unresolved status; escalation recorded | Success claimed without a receipt | Block release and investigate the unsupported status |
These are illustrative outcomes, not LuMay benchmark results. The second and fourth cases expose behavior that a polished final answer can hide. Fix the relevant prompt or application path, then rerun the affected scenarios and neighboring cases. Permission and approval enforcement still belong in the application boundary.
Keep a release gate that reviewers can inspect
A useful gate records the candidate version, dataset and evaluator versions, individual failures, the rule that blocked promotion, and the decision owner. If an exception is allowed, describe its scope, exposure, compensating control and expiry. Do not convert an unresolved critical failure into an exception simply to obtain a green pipeline.
For implementation choices, use LuMay’s AI agent testing frameworks for CI/CD. For a broken argument or adapter mapping, use the narrower tool contract testing guide.
Questions engineering teams ask about regression testing
Q1: Must a regression test require exactly the same answer?
A: Usually no. Different wording can satisfy the same task. Assert stable business requirements and use a rubric for acceptable response differences. Exact matching is useful only for fields whose exact value is part of the contract.
Q2: Can retrieval changes trigger regressions even when the prompt stays the same?
A: Yes. Different chunks, filters or ranking can change the evidence available to the agent. Version retrieval inputs and settings so the comparison can be explained.
Q3: How should a team handle a baseline that already fails?
A: Separate known failures from newly introduced ones. Record the existing defect and investigate test instability. Do not quietly redefine the expected result to make both releases pass.
Q4: Can a higher average score hide a worse release?
A: Yes. Improvements in many low-risk cases can conceal a critical failing case. Review case-level results and apply explicit rules for unacceptable effects.
Q5: What should happen when the evaluator changes?
A: Run the old and new evaluators on a shared set of outputs and inspect disagreements. A changed grade may come from the measuring method rather than the agent.
Q6: How can a team control evaluation cost?
A: Use a fast critical suite for routine changes and a broader suite at defined release points. Preserve high-risk checks and record the coverage excluded from each run.
Q7: Are mocked tools enough?
A: They support repeatable logic checks but cannot establish that a real integration behaves the same way. Add authorized sandbox checks for important mappings, permissions and persisted outcomes.
Q8: What if a model provider changes a model behind the same name?
A: Retain available model metadata and run scheduled checks against stable scenarios. A mutable alias limits reproducibility; record when the provider does not expose a fixed version.
Q9: Should every production conversation enter the test dataset?
A: No. Select representative, permitted examples and remove unnecessary sensitive data. Preserve the failure mechanism without indiscriminately copying production conversations.
Q10: How do you avoid tuning only to known tests?
A: Keep reviewed cases out of routine prompt tuning and add new cases from genuine usage. Passing a memorized suite provides limited evidence about unfamiliar requests.
Q11: Who should approve a release exception?
A: Name an accountable owner with the authority to accept the relevant business risk. Preserve the reason, affected workflows, conditions and expiry alongside the release record.
Q12: Does a passing suite prove the agent will never fail?
A: No. It establishes observed behavior under the tested conditions. Maintain monitoring and an incident process for failures the suite did not represent.
Start with one workflow that must keep working
Choose an existing workflow, record its accepted behavior and compare the next release against it. A useful result tells the team which requirement changed, what evidence demonstrates it, and who decides whether the change can ship.
Bring a solution you want to productionize, together with one workflow that must remain reliable. LuMay’s AI engineering services provide a starting point for discussing integration, testing and release requirements.





