Measure resolved chatbot requests and remaining work

Microsoft distinguishes confirmed and implied resolutions, escalations and abandoned sessions in Copilot Studio. One conversation can generate several analytics sessions. Depending on the flow, a timeout can count as implied success; triggering a handover also does not prove that an employee accepted it. These distinctions appear in the outcome measurement documentation and monitoring definitions, checked on 26 September 2026.

Do not adopt a vendor metric as your resolution rate without checking its definition. Record the numerator, denominator, period, exclusions and source for every measure. Separate billable vendor events from independently reviewed business outcomes: a billing unit primarily determines charges.

The table defines six chatbot quality metrics (verified completion rate, critical error rate, successful handover, repeat-contact rate, handling effort and customer satisfaction) and the decision each supports.

No single metric replaces the others. A low escalation rate is undesirable when the bot supplies false answers instead. High satisfaction among a handful of voluntary respondents does not prove good performance for everyone. Report counts and selection methods alongside percentages.

Data. A fictional month contains 1,000 eligible customer requests. The bot labels 700 complete. A subject review of the entire example dataset confirms 630; the remaining 70 are uncertain or incorrectly closed. Another 250 requests are handed to the team, and 50 end without an outcome.

Labelled versus verified. The automatically labelled completion rate is 70 percent. The verified completion rate in this example is 63 percent. Both calculations are correct, but they describe different evidence. If only 100 cases had been sampled, the other 900 could not be described as individually verified.

Handover states. Of the 250 requested transfers, 240 reach the correct queue and 220 are accepted within the internally agreed time. This produces three distinct states: requested, received and accepted on time. Counting tickets alone would conceal the last distinction.

Rubric. Define a good answer before judging conversations: correct information, appropriate source, relevant country or product variant, authorised data access and a correct next step. Score those dimensions separately. A friendly answer can be factually wrong; a cautious clarification can be the right response.

Critical errors. Give critical errors their own category. Examples include another customer's order data, invented refund confirmations or binding commitments without approval. They must not disappear statistically among many correct opening-hours answers. Establish an accountable owner and a suspension process for affected capabilities.

Automated evaluators. Automated evaluators can triage conversations or flag possible source conflicts. Calibrate their judgments against human-reviewed examples. When the bot and evaluator rely on the same misleading knowledge entry, both may consider the same false answer plausible.

Selection. Do not select only successful conversations or the longest chats. Include common tasks, uncommon high-risk cases, languages, devices, abandoned attempts and escalations. Record whether selection was random or deliberately stratified by risk. A risk-focused sample is useful for finding faults, but it is not a representative overall rate without suitable weighting.

Two reviewers. Have two reviewers assess disputed cases and reconcile their judgments. This improves the rubric instead of treating disagreement as noise. Retain only information needed for evaluation; pseudonymous case references are often more useful than complete customer profiles.

Regression set. Keep a stable regression set for known failures and a separate set of previously unseen cases. Testing only the examples used to tune the bot can overstate how well the correction generalises.

Include these items in the time and cost of a complete request:

• bot operation
• knowledge maintenance
• quality review
• residual handling
• repeat contacts

When an automatic answer triggers an additional phone call, that work belongs in the outcome. So does the time staff spend repairing unusable summaries.

Comparable periods. Compare similar request types and periods. A season dominated by straightforward shipping questions is not a fair baseline for a later wave of complaints. Record changes to bot versions, sources, opening hours and channel access. Otherwise an apparent improvement may reflect a different case mix.

Median and slow cases. The median describes a typical duration, but inspect slow cases and unresolved queues as well. A quick first message can conceal a long time to actual resolution. Distinguish waiting for customer information from waiting for an internal action so the responsible team can make the relevant improvement.

Review several unsuccessful cases weekly with the service owner. Classify their causes:

• missing knowledge
• incorrect retrieval
• unclear clarification
• permission failure
• system outage
• an unstaffed queue

Each cause needs a different correction; a longer prompt will not resolve all of them.

Approve the next version against the known cases and fresh examples. Record which change is intended to fix which fault. A useful chatbot dashboard then supports reviewable operating decisions and makes clear where the available evidence remains insufficient for a reliable conclusion.

These pages cover the service overview and the related decisions for measuring chatbot quality:

• AI chatbots for customer service
• Customer-service AI chatbots: plan the human handover
• Launch and operate a chatbot with clear release criteria
• Compare chatbot costs per useful outcome

For a workflow review, start with the task, systems involved and typical exceptions. Discuss it with modernagents.