PRACTICAL GUIDES
Air Canada’s Chatbot Got the Policy Wrong. Would Yours?
Explore the Air Canada chatbot case and the 2024 ruling, with practical lessons for testing AI customer-support answers against refund policies and exceptions.
The short answer
In Moffatt v. Air Canada (14 February 2024), British Columbia’s Civil Resolution Tribunal found that Air Canada’s chatbot gave misleading advice about retroactive bereavement fares. The customer had relied on it when booking in November 2022. The lesson for AI customer support: a policy link is not proof that an answer follows the policy. Test eligibility, deadlines, exceptions and promised actions against the approved rules.
The incident: a helpful answer with an expensive contradiction
This is a historical case, not a new incident. In November 2022, Jake Moffatt used Air Canada’s website chatbot while arranging travel following a family bereavement. The chatbot said a traveller could submit a ticket for a reduced bereavement rate within 90 days of its issue, including after travel. The linked policy page said the policy did not apply to requests made after travel had been completed.
Moffatt relied on the chatbot’s information when booking. On 14 February 2024, British Columbia’s Civil Resolution Tribunal found negligent misrepresentation and ordered Air Canada to pay CAD $812.02: $650.88 in damages, $36.14 in pre-judgment interest and $125 in tribunal fees. This was a tribunal award, not a regulatory fine.
The decision records that Air Canada supplied no information about the nature of its chatbot. It does not establish a particular model, retrieval architecture or technical root cause. Calling this a proven large-language-model hallucination would go beyond that evidence. The lessons below apply to modern AI assistants, without assuming how this earlier chatbot worked.
The important detail: the answer already had a link
The chatbot response linked to the bereavement policy. That did not resolve the contradiction in its own advice. The tribunal questioned why a customer should have to double-check information on one part of the airline’s website against another.
For a business, this separates three tests that often get bundled together. Did the assistant retrieve an approved source? Did the source actually support the answer? And did the answer preserve the conditions that change what a customer should do next?
A reply can pass the first check and fail the next two. “You may qualify” is not the same as “you will receive a refund.” “Apply before travel” is not the same as “apply within 90 days.” A polite tone and a genuine link do not repair an unsupported promise.
A practical exercise: test the promise, not just the paragraph
Use a fictional retailer with an approved rule: refund requests must be submitted within 30 days of delivery, and opening a request does not guarantee approval. These are invented rules for a staging exercise, not Air Canada’s policy.
Ask “Can I buy now and get my money back later?” Then vary one condition at a time: the delivery date is unknown, the request arrives on day 30 or day 31, an old FAQ says 60 days, or the customer insists a previous agent promised an exception. Define the expected answer with the policy owner before testing.
Check the actual response for invented eligibility, missing conditions and unsupported guarantees. If the assistant can open a refund request, inspect that record too. The customer should not receive “refund approved” when the system has only created a case for review.
- 1. Fix the source of truth
Record the approved policy version, effective date, owner and treatment of exceptions. Decide which source wins when a FAQ and the policy disagree.
- 2. Change the customer’s situation
Test boundary dates, missing facts, different wording, repeated questions and claims of prior approval. Include valid requests so a system that refuses everything cannot appear successful.
- 3. Inspect the promise and the action
Compare each material statement with the applicable rule. Where tools are involved, compare the answer with the submitted fields and final record, not only the tool’s success response.
- 4. Recheck after changes
Keep failed cases and rerun them after policy, prompt, model or integration changes. Record attempts and conditions; one correct answer does not establish reliability.
What a useful handover looks like when the answer is uncertain
When the evidence is missing or contradictory, agree a response that identifies the uncertainty without inventing a favourable outcome. For the fictional retailer, that might mean asking for the delivery date or sending an exception request to the authorised team. It should not mean guessing eligibility to keep the conversation moving.
Test the handover as a workflow. Did the correct team receive the unresolved question, relevant facts and conflicting sources? Can the customer see what happens next? If the handover fails, does the assistant acknowledge that failure instead of saying someone will be in touch?
The goal is useful assistance within the approved rules, not blanket refusals or a disclaimer attached to otherwise misleading advice.
Where VisiFlo could help
VisiFlo can help scope a focused assessment of one customer-support workflow: agree the business rules, turn them into realistic test cases, inspect supported answers and actions, and document failures and evidence gaps. The useful output is a prioritised list of what to fix and what to recheck, tied to the customer decisions at stake.
For a retailer, that could be returns and warranty eligibility. For a booking business, cancellation and rescheduling. For a service company, quotes, inclusions and promises about turnaround. Start with the answers customers are most likely to rely on before spending money or giving up another option.
VisiFlo did not assess Air Canada, and this article does not claim we could have prevented the incident. Testing provides evidence for defined cases, versions and observable stages, not a guarantee against every failure. Access, scope, price and timing are agreed before work starts; engineering fixes are separately scoped.
Bring one policy and one customer journey. We can discuss which promises deserve testing before your assistant makes them in public.
Common questions
What is chatbot policy validation?
Chatbot policy validation checks whether customer-facing answers and actions follow the business’s approved rules. It tests eligibility, deadlines, exceptions and handovers against a known policy version. Finding a relevant source is only one part of the check; the final answer must preserve the conditions that affect the customer’s decision.
Was the CAD $812.02 payment a fine?
No. In Moffatt v. Air Canada, the tribunal ordered CAD $650.88 in damages, $36.14 in pre-judgment interest and $125 in tribunal fees, totalling CAD $812.02. It was an award in this small-claims dispute, not a regulatory fine.
Would adding a policy link have prevented this issue?
The chatbot already linked to a policy page. The lesson is to test whether the answer follows the source, including eligibility, timing and exceptions, rather than treating the presence of a link as proof of correctness.
Does the decision prove a generative AI model caused the error?
No. The decision says Air Canada did not provide information about the nature of its chatbot. It establishes the misleading advice and the tribunal’s findings, not a model or technical root cause.
Is this a legal rule for every Australian business?
This was a British Columbia small-claims decision on specific facts. The article offers workflow-testing guidance, not legal advice or a conclusion about Australian liability. Ask qualified counsel about your own obligations.
What should we measure in a policy-answer test?
Track unsupported promises, omitted conditions, correct handling of valid requests, unresolved cases and handover results. Preserve the policy version, inputs, outputs and observable actions. Report the number of runs and any stages you could not inspect.
References and further reading
The guidance above is VisiFlo’s interpretation for business testing. These sources provide context for reported events, platform behaviour and risk concepts.
Make your next AI decision with clearer evidence.
Bring one workflow, idea or unexpected result. We agree fit, scope, price and timing before work starts.
Discuss your challenge ↗Explore AI validation and testing services →