On this page
A support bot answers 200 questions a day in clean, confident sentences. Almost nobody complains. The dashboard shows a 78 percent containment rate, so leadership calls the rollout a success and moves on to the next project.
Then a customer acts on a shipping window the bot invented, a refund goes out that never should have, and support spends a week untangling it. The containment number and reality had quietly drifted apart. The team had been measuring how often the bot replied without a human. Whether the bot was right is a separate question, and it needs separate instruments.
This guide covers what chatbot accuracy actually means, how to measure resolution rate honestly, what a good benchmark looks like, how to build a test set that scores answers, how to catch wrong answers in production, and how to raise the number over time.
What does chatbot accuracy mean?
Chatbot accuracy is whether an answer is understood correctly, factually right, complete, and routed to the correct next step, judged against your own sources rather than the customer's mood. It is a property of the answer, and it breaks into parts you can inspect one at a time.
Glean's teams split accuracy into four components: intent accuracy (did the bot grasp what the visitor wants, past the exact wording), answer correctness (is the information factually right and consistent with your current policy), completeness (did the reply cover every detail needed to resolve the question), and action accuracy (did it route, escalate, or trigger the right workflow). A reply can pass the first check and fail the last one, which is why a single thumbs-up rating hides more than it shows.
Two numbers get confused with accuracy and should be kept apart from it. Customer satisfaction, or CSAT, measures how a person felt about the interaction, and a customer can rate a chat five stars while walking away with a wrong answer they have not discovered yet. Containment measures how often the bot handled a chat without a human, which counts effort rather than correctness. Both are useful outcomes to report to the business. Neither one tells you whether the words the bot produced were true.
Grounding is what makes accuracy checkable in the first place. An AI chat assistant that answers from a curated knowledge base can be measured against its sources, while a bot answering from the model's memory alone gives you nothing to check the answer against.
How do you measure resolution rate?
Measure resolution rate by tagging every conversation as resolved, escalated, or abandoned, dividing genuinely resolved chats by total chats, and pairing the result with recontact rate so a customer who quietly gave up never counts as a win. The tagging is the hard part, because the default definition most tools ship is generous to the bot.
Many platforms mark a chat resolved the moment the visitor stops replying. Silence is read as success. Some of those silent visitors did get their answer. Others got frustrated and left, and a few will be back tomorrow asking the same thing. Glean's guidance is to read resolution alongside recontact rate, the share of resolved visitors who return within a set window with the same issue, so the abandoned cases surface instead of hiding inside the headline.
This is where containment and resolution split. Containment counts chats the bot kept off human hands. Resolution counts chats where the problem actually got solved. A store can post a flattering 70 percent containment while its true resolution sits near 50, because a fifth of those contained chats ended with an unhappy person who simply closed the tab.
Here is the arithmetic on one month. A site logs 2,000 bot conversations. The tool reports 1,400 contained, a tidy 70 percent. Audit those 1,400 and you find 240 visitors recontacted within seven days with the same question, while another 120 abandoned the chat mid-thread without an answer. Subtract the 360 that were never really solved and you are left with 1,040 genuine resolutions, a 52 percent resolution rate. Same month, same logs, an 18 point gap between the number that flatters and the number that is honest.
What is a good accuracy benchmark for a chatbot?
A solid steady-state resolution rate for most support and sales bots runs about 40 to 60 percent, best-in-class deployments reach 60 to 75 percent, and a well-grounded bot should keep wrong answers rare enough to measure in low single digits. Those numbers give you a target, but read them against your own traffic, because a bot answering narrow order-status questions and one fielding open product questions live in different worlds.
Benchmark roundups of 2025 support data put resolution at a 40 to 60 percent steady-state target, with 60 to 75 percent marking best-in-class and usually requiring a maintained knowledge base plus several months of tuning. Containment on a well-configured retrieval bot lands higher than that, which is exactly why the gap between the two numbers is worth watching rather than celebrating.
For factual accuracy, the cleanest public reference is Vectara's hallucination leaderboard, which measures how often a model adds unsupported claims when it summarizes a source document. On that test the strongest models in late 2024 stayed under 5 percent, with Google's Gemini 2.0 Flash near 0.7 percent. Treat that as a ceiling rather than a promise. The leaderboard uses short, clean documents, while your production traffic brings messy phrasing, missing context, and multi-turn threads, all of which push real error rates above what a tidy benchmark suggests. The benchmark tells you what is possible on an easy task. Your own test set, covered next, tells you what is happening on yours.
How do you build a test set to score accuracy?
Build a golden test set of real questions paired with approved answers, then score the bot's replies against it with an LLM judge on every release, so accuracy becomes a number you gate on rather than a feeling. This is the offline half of measurement, the regression test you run before a change ships, and it catches breakage that live monitoring would only find after customers do.
Start with the questions your sales and support teams actually get, pull 100 or more, and write the correct answer for each straight from your own docs. For each reply, the two workhorse scores are faithfulness, whether the answer stays inside the retrieved sources, and answer relevancy, whether it addresses what was asked. Both require reading comprehension, so they are graded by an LLM acting as judge rather than by string matching, an approach DeepEval and other frameworks build directly into their metrics.
The judge is reliable enough to trust as a gate. Evidently's guide to the method notes that a well-prompted LLM judge agrees with human reviewers roughly 80 percent of the time, about the level at which two human reviewers agree with each other on the same call. Automating the scoring is what lets you run the full set on every change instead of once a quarter.
A worked example makes the cadence concrete. Say you keep a 120-question golden set and run it automatically each night, with a rule that any release scoring below 0.90 on faithfulness gets blocked. A prompt tweak meant to make answers friendlier quietly drops faithfulness from 0.91 to 0.86. The gate catches it overnight, the change never reaches a customer, and the AI assistant setup rolls back to the prompt that scored clean. Without the test set, that same regression ships on Friday and you learn about it from complaints on Monday.
How do you catch wrong answers?
Catch wrong answers by scoring every live reply for groundedness and model uncertainty, checking that each claim maps to a real source passage, and having a person read a small random sample plus everything the automated scores flag. Offline tests prove the bot works on questions you thought of. Production monitoring catches the ones you did not.
Three checks can run automatically on live traffic. Groundedness scoring reruns the faithfulness idea on real answers, flagging or holding any reply that drifts past its sources. Citation matching goes claim by claim and holds anything that cannot be tied back to a retrieved passage. Uncertainty scoring flags answers the model itself is shaky on: the Nature study on semantic entropy by Farquhar and colleagues showed that measuring a model's uncertainty at the level of meaning, rather than exact wording, reliably detects the confident fabrications the authors call confabulations, and it generalizes to tasks the method was never tuned for.
Automation has a ceiling, so a human closes the gap. Sample a small slice of daily chats, read them against the sources, and always review whatever the scores flagged. Then add the passive signals your product already collects: a thumbs-down, a fast re-ask of the same question, a spike in recontacts on one topic. Each is a pointer at an answer worth reading.
Put numbers on it. A bot handles 500 chats a day. You audit a random 2 percent, 10 chats, plus the 6 that the groundedness score flagged, so a reviewer reads 16 conversations against source. Find two ungrounded answers among the flagged set and you have both a rough error rate to track and two concrete gaps to fix, long before those errors show up as an angry email.
How do you improve accuracy over time?
Improve accuracy by turning every miss into content: log the questions the bot flubbed, close the knowledge gaps behind them, re-run your test set to confirm the fix held, and repeat on a fixed cadence with one person owning the loop. A bot does not get more accurate on its own. It gets more accurate because someone feeds its failures back into the source material.
The mechanism is a feedback loop. Every flagged answer, every unresolved question, and every ticket a human had to finish points at a gap in the knowledge base. Teams that route those gaps into new or corrected content, then re-index, report bots climbing from roughly 50 to 70 percent of questions handled at launch to 80 percent and up within a few months. The model did not learn anything new. The content it reads got better.
Keep the loop honest with two habits. Re-run the golden test set after every content change so you can see whether a fix helped or quietly broke an answer that used to work, and keep a human in the loop for the sensitive topics where a wrong answer is expensive. When a question sits outside the bot's knowledge, or a high-intent buyer clearly needs a person, the right move is a fast handoff to live chat instead of a confident guess.
A worked example ties it together. Month one, resolution sits at 48 percent. You pull the ten topics that drove the most misses, write or fix the content behind each, and re-run the test set to confirm nothing regressed. Repeat monthly. By month three the same bot resolves 63 percent. The model never changed. Three rounds of gap-closing sharpened what it reads, and the measurement loop proved each round paid off.
Key takeaways
- Accuracy has four parts. Intent, correctness, completeness, and routing can each fail on their own, so inspect them separately rather than trusting a single thumbs-up.
- Resolution rate beats containment. Containment counts chats the bot kept off human hands, so pair resolution with recontact rate and a customer who gave up never gets filed as solved.
- Target 40 to 60 percent resolution, with wrong answers in low single digits. Best-in-class reaches 60 to 75 percent, and the strongest grounded models hold hallucination under 5 percent on clean benchmarks.
- A golden test set turns "seems fine" into a score. A hundred or more real questions with approved answers, graded by an LLM judge for faithfulness and relevancy, catch regressions before customers do.
- Catch wrong answers with layered checks. Groundedness and uncertainty scoring on every chat, citation matching, and a human sample audit find the fabrications that automated scores alone miss.
- Improvement is a content loop. Log misses, fill the knowledge gaps behind them, re-run the test set, and repeat on a cadence with one owner, and resolution climbs month over month.
TAGS

Written by
Nilas MylerCo-founder & CTO, Glimpze
Nilas is the co-founder and CTO of Glimpze, an inbound sales tool that turns high-intent website visitors into live conversations. A former SEO consultant for some of the largest companies in Denmark, he writes about speed-to-lead, inbound sales, and conversion rate optimization — the technical and operational mechanics of turning traffic into pipeline.