How to Test AI Agent Failures, Retries and Recovery.

Start with an uncertain outcome

In this invented example, a refund service accepts an AUD 240 refund, but its confirmation times out. The agent cannot tell from the timeout alone whether the refund happened. Retrying as a new operation could issue money twice.

An operation identifier, sometimes called an idempotency key, lets a service recognise a repeated request. Support, expiry and retry behaviour depend on the actual service. A model instruction alone does not enforce this protection.

Five recovery cases worth agreeing

1. Timeout after an action: confirm whether it completed before attempting a new payment, booking or message.
2. Repeated request: replay the same event and inspect whether the system creates a duplicate action.
3. Unavailable tool: verify that the agent reports an incomplete task instead of claiming success.
4. Changed or expired approval: change the order, amount or requested slot while approval is pending. Check that stale permission is rejected.
5. Partial completion: complete one step and fail the next. Record the actual state and the agreed recovery or human handover.

Check state, not just the conversation

For each case, retain the workflow version, starting records, request, injected failure, action attempts, results and final state. Distinguish an expected failure that was handled correctly from a test that could not run. A friendly response cannot establish whether an external record changed.

Repeat relevant cases after a prompt, model, tool or retry-policy change. Use explicit checks for forbidden actions and approval matching; use human review for explanations that require judgement.

Give the client a useful next action

Report the rule, finding, supporting evidence, owner and next action. If a failure can be reproduced, agree a regression check and repeat it after the correction. If it cannot, retain the limitation rather than reporting a pass.

Use the sample report and editable checklist to structure the handover. See also the refund approval checks and client handover guide.

Agree a controlled scope

Use synthetic or appropriately redacted records and an authorised sandbox. Agree which failures can be simulated, which state can be inspected and what cleanup is required. This guide describes test design; it does not document a completed VisiFlo integration or customer evaluation.

A finite suite supports a scoped release decision. It does not guarantee future behaviour or replace a security, privacy or compliance assessment.