How to check AI output before it reaches a customer

The first month after a firm switches on AI-drafted client emails, somebody reads every one before it goes out. By month three, nobody does, and nobody decided that. The reviewer got busy, the drafts looked fine, and the checking quietly stopped. That is the normal way review dies, and it is the wrong way. Most of the checking should have been done by the system before a person ever opened the draft. This article covers how to check AI output quality in a company with no quality department: the four checks, the automated gates that run before a human looks, and the scoring log.

Quick Answer: Check AI output quality in layers: rule-based pre-checks on every output (commitments, required fields, numbers traced to source), a second AI pass that can hold but never approve, and a human queue scored pass or fail on four checks.

What should you check in AI output before it reaches a customer?

Check four things, in this order: whether every fact is true, whether the output commits your firm to anything (a price, a deadline, a legal position, a refund), whether tone and scope fit the customer, and whether a required step is missing. Most review only checks the third, because tone is what a fluent draft makes visible.

The order matters because the damage runs in the same order. A wrong fact costs a correction. A wrong commitment costs money. In February 2024 the British Columbia Civil Resolution Tribunal found Air Canada liable for a chatbot that told a grieving customer he could claim a bereavement fare after travelling. The airline argued the chatbot was a separate legal entity responsible for its own words. The tribunal rejected that and held the company responsible for all information on its website, chatbot included. The damages were under a thousand Canadian dollars. The precedent was the expensive part.

Each check has a different best checker, and that is the design principle behind everything below. Rules can flag three of the four without judgment (a commitment word, a missing field, a number or name that does not match the source record). A second AI pass grades the fourth, tone and scope. What no gate can do is decide: whether a flagged commitment is allowed, and whether the source a number was traced to is the right one. That is the human's job, and it is a fraction of the reading.

1. Facts: every name, number, date, product detail, and policy claim is traced to a source the reviewer can open. A system can do the tracing (does this number appear in the source record?). Only a person can confirm the source is the right one.
2. Commitments: anything a customer could later hold you to. Prices, timelines, "we will", "you are entitled to", "no charge", "guaranteed". A rule can catch every one of these words. A person decides whether the commitment is allowed.
3. Tone and scope: does the output answer the question asked, at the right length, for this customer, without unsolicited extra advice that widens what you are accountable for? A second AI pass is good at this. It is the one check where AI grading AI works well.
4. Missing steps: a disclaimer, a signature line, a case number, a next action. A rule catches this perfectly, because the requirement is fixed and the check is "present or absent".

Why can't you skip reviewing AI output?

Because the error rate of current AI tools is high enough to matter and low enough to lull you. Purpose-built tools from vendors who promised accuracy still invent facts a meaningful share of the time, which is why an AI vendor risk assessment tests the accuracy claim before you sign. Customers rarely tell you when it happens, and courts treat the output as yours. Review is the cost of using the tool.

A Stanford RegLab study published in May 2024 tested the legal research tools from LexisNexis and Thomson Reuters and found they hallucinate (invent false information) between 17 and 33 percent of the time, despite both vendors marketing them as hallucination-free or close to it. Those are premium tools built for a single profession. A general chat assistant drafting your client emails is not held to a higher standard.

Damien Charlotin's database of court decisions where a party relied on hallucinated content listed more than 1,900 cases as of late August 2026, over 1,300 of them in the United States. Those are the cases that reached a judge. Most AI mistakes never do; they reach a customer.

And customers mostly stay silent. Qualtrics' 2026 Consumer Experience Trends Report, based on a Q3 2025 survey of more than 20,000 consumers in 14 countries, found that 47 percent of bad experiences lead to reduced spending, while only 29 percent of customers tell the company directly, down from 36.5 percent in 2021. You will not learn about your AI's mistakes from complaints. You will learn from the revenue line, months later, unless you check first.

What should the system check before a human sees the output?

See your next AI opportunities in 3 minutes.

Your likely bottlenecks, and the AI solutions worth doing next.

Get my AI roadmap

Three automated gates, run on every output in this order: rule-based pre-checks that need no judgment, a second AI pass graded against the same four checks, and a router that sends each output to one of three lanes (auto-send, review queue, or block). The gates cannot approve anything. They decide which outputs a human must see, and they log everything.

Gate one is rules. Rules are cheap, deterministic, and never get tired, so they take the checks that have a yes-or-no answer.

Gate two is a second AI pass. Give a separate AI tool the draft, the source record, and the four checks, and ask for pass or fail on each with a one-line reason. It is good at tone and scope, useful on facts as a second opinion, and it catches things rules miss. Two rules govern it: it can hold an output but never approve one, and any disagreement between it and the rules gets treated as a hold. AI grading AI as the final gate fails because two tools trained on similar data share blind spots, and courts treat the output as your words regardless of how many machines looked at it.

Gate three is the router. Every output that survived gates one and two lands in one of three lanes. Auto-send: nothing fired, the workflow has met its exit criteria, and this output was not picked for the sample. Review queue: something fired as a hold, or the output was picked for the sample, or the workflow is still in its 100 percent phase. Block: a required field is missing, the recipient does not match, or a commitment appeared in a workflow that is not allowed to make one. Blocked outputs go back to the draft step, never to the customer.

The whole stack runs inside the automation that produces the draft. A firm on a standard workflow tool can build it as steps between "draft generated" and "email sent", with the review queue living in the inbox, ticketing system, or document tool the reviewer already uses. Nothing about it requires a separate quality platform.

How do you score AI output quality?

Score each reviewed output pass or fail on the four checks, then record one of three outcomes: sent as-is, sent after a light edit, or blocked. The outcome, the failed check, and a one-line reason go into the log. Thirty seconds per review turns opinion into an error rate you can act on.

Avoid one-to-ten scales. A seven means nothing when two reviewers disagree on what a seven is. Pass or fail per check, plus a named reason for each fail, survives handover to a second reviewer and produces a number at the end of the month.

The AI implementation checklist for small business treats a pilot as a 30 to 60 day test with one metric. For a customer-facing workflow, the blocker rate is that metric.

Three refinements make the log worth more than the thirty seconds it costs.

First, capture the edit itself, rather than the fact that an edit happened. When the reviewer changes a draft before sending, the system stores the before and after. A month of those pairs is the best evidence you will ever have for rewriting the tool's instructions, because it shows you exactly which sentences the AI gets wrong and how a person fixes them.

Second, separate the reviewer's verdict from the machine's. The log should hold what the pre-checks said, what the second AI pass said, and what the human decided, as three fields. When the human overrules the machine, that is a case worth reading. When the machine flagged something the human waved through, that is a case worth reading twice.

Third, measure the reviewers. Once a month, route ten already-reviewed outputs to a second reviewer without telling either person. If the two disagree on more than one in ten, the checks are not defined tightly enough, and your blocker rate is measuring reviewer mood as much as AI quality.

Where does AI output review go wrong?

Three ways, and the first is the one nobody plans for: reviewers stop seeing errors. Automation bias is a documented effect, not a discipline problem. Parasuraman and Manzey's 2010 review in Human Factors found that over-reliance on automated aids occurs in novices and experts alike and cannot be prevented by training. It worsens when the reviewer is juggling other tasks.

That describes every office manager reading AI drafts between phone calls. The defence is structural, and it is the reason the rules and the second AI pass exist: they take the checks a tired human skips. The system also makes it hard to skip the checks that remain with the human. The reviewer checks facts against the source document, which the queue shows side by side with the draft. The log forces a per-check verdict rather than a general impression. And the review task gets a fresh reader periodically, because the person who has read 400 clean drafts is the least likely to catch the 401st.

The second failure is reviewing the wrong thing. A skilled writer reviews for tone because tone is what they notice. Fabricated references pass because they look like references. In July 2025, Deloitte's Australian arm published a report for the Department of Employment and Workplace Relations under a contract worth AU$440,000. It contained references to nonexistent academic papers and a fabricated quote from a federal court judgment. A university researcher found the errors after publication. Deloitte repaid the final installment and issued a corrected version in October. Many people had read that report before release. Nobody opened the citations. A number-tracing pre-check would have held every one of those references on the first pass.

The third failure is the opposite: 100 percent review that never ends, which quietly deletes the return on the automation. If a partner reads every AI-drafted client letter for twelve months, the firm has bought a slower typist. Every AI automation audit we have run for a law firm has surfaced ten or more automation opportunities. When those get ranked, the review burden is part of the score. A workflow that a paralegal can sample ranks above one that a partner must read in full. The usual fix is to shrink the AI's job until sampling becomes safe, not to keep a senior person reading forever.

There is one case where the right answer is not to automate yet. If the source data the AI draws from is itself unreliable, no review process fixes the output, because the reviewer and the pre-checks are both checking the draft against the wrong source. Fix the data first. A breakdown of what law firms can automate covers which client-facing workflows tend to be ready and which are not.

---

Not sure which of your customer-facing workflows could run on a sample and which need a human on every output? The free AI assessment sorts that out. Six questions, two minutes, and you get a personalised preview of where AI would pay off in your firm and what it would take to keep it safe. Start your free AI assessment.

Frequently asked questions

How do you check AI output quality without reading everything?

Let the system do the checks that have a yes-or-no answer, then read a sample of the rest. Rule-based pre-checks catch commitments, missing required fields, numbers with no source, and wrong recipients on every output. A second AI pass grades tone and scope and can hold a draft but never approve it. Human reviewers score the remaining outputs pass or fail on four checks and log the result. Once a case type has produced 100 outputs in a row with zero blockers, you can claim a blocker rate below 3 percent, and the router switches that type to a random sample plus every risk-flagged output.

What is an acceptable error rate for AI output?

It depends on what an error costs, and for some outputs the only acceptable rate is zero. For outputs that commit your firm to money, timelines, or a legal position, no error rate is acceptable, so those stay under full human sign-off or get redesigned so the AI never produces the commitment. For correctable customer-facing outputs, a sensible target is a blocker rate below 1 to 3 percent, which takes 100 to 300 consecutive reviewed outputs with no blockers to demonstrate. For internal drafts, higher rates are tolerable because the reader is a colleague who will catch the mistake.

Who should review AI output in a small business?

The person who already owns the workflow and knows the source documents, not the most senior person available. A reviewer checks facts against the underlying record, so the receptionist who knows the booking system is a better reviewer of appointment emails than a partner who does not open it. Reserve senior review for the outputs that bind the firm. Rotate reviewers periodically and double-review ten outputs a month, because research on automation bias shows that over-reliance affects experts as much as novices and is not fixed by training.

Can AI check the output of another AI?

Yes, as a gate that can hold but never approve. A second AI tool given the draft, the source record, and the four checks is good at tone and scope and useful as a second opinion on facts, and it cuts the number of outputs a person must read. It should never be the final check, because two AI tools trained on similar data share similar blind spots, and courts and customers treat the output as the company's own words regardless of how many machines looked at it. Treat any disagreement between the AI grader and the rule-based checks as a hold for human review.

What should the AI output review log contain?

Per output: what the pre-checks said, what the second AI pass said, and what the human decided, as three separate fields, plus the outcome (sent as-is, light edit, or blocked), the failed check, a one-line reason, the case type, and the before-and-after text of any edit. From those fields you get the blocker rate per case type, which decides whether review can be reduced, the light-edit rate, which shows where the instructions or template need work, and the overrule cases, which are the ones worth reading each month. Keep the log inside the system the reviewer already uses.

Does reviewing AI output cancel out the time savings?

Only if review is designed badly or never scaled down. When rules and a second AI pass handle the yes-or-no checks, the human reads a fraction of the outputs and spends about thirty seconds on each, against the ten minutes the draft would have taken to write. The savings disappear when a senior person reads every output indefinitely, or when review is a vague read-through that catches tone and misses facts. The fix is a defined check, a log, and written exit criteria for stepping the sample rate down. If a workflow can never safely move to sampling, shrink the AI's role rather than keep paying for full review.