Containment Is Not Resolution: A Case-Level Chatbot Scorecard

The decision: measure the customer’s issue, not the chat window

A chatbot conversation ending is an event in your software. It is not proof that the customer’s problem ended.

A useful service scorecard must follow the issue far enough to detect a handoff, a failed action, a return on another channel, and an outcome the customer can verify.

If your team is reviewing a Swiss SME chatbot pilot, the existing implementation guide for Swiss SME customer-service chatbots is a useful deployment baseline; this article focuses on the narrower operating question that follows deployment: what evidence is enough to call an issue resolved?

The practical recommendation is to report containment and verified resolution as different measures, attach them to a case or issue episode rather than a vendor session, and publish both numerator and denominator. Keep a human route visible.

A bot may contain a session while the customer abandons it, tries a second channel, or repeats the same question the next morning. Those outcomes can look successful in a dashboard built only from bot events.

This is an operating protocol, not a claim that every SME needs a sophisticated contact-centre data platform. A small team can start with a ticket tag, a documented time window, and a weekly manual sample.

The important part is deciding what counts before results are shown to management. Changing the definition after seeing the rate creates a metric that follows the product’s preferred story instead of the customer’s outcome.

Why the dashboard can be right and still mislead

“Containment” generally answers a channel question: did the conversation leave the automated channel for a person? “Resolution” asks an outcome question: was the underlying issue addressed correctly and completely? These are not interchangeable.

A conversation can be contained because the customer got a correct answer, because the customer gave up, or because the escalation control was difficult to find. The same status - closed - can represent all three.

The difference matters economically as well as experientially. A bot that ends a chat without solving the problem may push work into email, phone, a store, or a second chat. The work has moved; it has not necessarily disappeared.

Conversely, an appropriate escalation can be a good customer outcome when a human needs authority, context, or judgment. Counting every handoff as failure can reward a system for refusing to escalate when it should.

The U.S. Small Business Administration lists a website chatbot as one possible way to answer common questions or complete an order, while also advising small businesses to review AI output and consider risks. That is general US small-business guidance, not Swiss legal advice, and it does not establish a benchmark. Its useful implication here is modest: select a bounded task, test whether it helps, and keep a person responsible for the customer-facing result.

SBA small-business guidance

What the evidence says, and what it does not say

An unresolved question that appears in real support discussions is whether a closed conversation actually means a customer issue is finished. A Power Platform community member described a bot repeatedly returning its previous answer, including for unrelated questions. That is an anecdotal troubleshooting post, not a representative sample, but it exposes a concrete failure mode: a bot can emit a plausible reply and end the interaction without advancing the customer’s task.

Power Platform Community thread on repeated chatbot responses

For measurement design, the U.S. Census Bureau’s statistical-quality material is a useful methodological reference rather than a chatbot source. It distinguishes sampling error from non-sampling problems such as coverage, nonresponse, measurement, linkage, and processing errors, and it calls for uncertainty to be considered when drawing conclusions from samples. A small SME transcript review is usually not a probability sample, so it should not be dressed up with a population confidence interval. Use the reference to make missingness, matching coverage, and reviewer selection visible.

Census Bureau Statistical Quality Standard E1

Bain’s service-centre brief describes predictive routing and first-contact resolution in large service operations. It illustrates that routing and the capability of the receiving agent matter to service outcomes. Its cases concern particular enterprise operations and a routing intervention, not a general chatbot containment target. This distinction is useful: an AI system can create value by helping the right human solve an issue, even when automation did not finish the issue alone.

Bain, AI Is Lifting Service-Center Performance

A recorded Call Centre Helper webinar on first-contact resolution explicitly covers how to define and measure FCR, and how not to use it. The agenda itself is a reminder that even a familiar service metric depends on scope and incentives. A video or webinar agenda is practitioner material, not an independent estimate of chatbot outcomes. The recommendation below therefore uses an explicit local definition instead of importing a headline target.

Call Centre Helper webinar: Innovative Best Practices for First Contact Resolution

A practical vocabulary for one issue episode

Start with an issue episode: one customer need, potentially represented by multiple conversations, channels, and tickets within a defined window. An episode is not necessarily identical to a vendor’s session identifier.

If a customer asks about the same delayed parcel in chat and then calls, those are two interactions but may be one unresolved issue episode. If the same customer later asks about a different order, that is likely another episode. Your matching rule should be documented and conservative where identity or topic is uncertain.

Use separate outcome labels.

• Resolved by automation means the bot’s action or answer meets a task-specific completion rule, with no qualifying repeat contact or known reversal during the observation window.
• Resolved after assisted handoff means a person completed the task, and the bot transferred enough context to avoid making the customer restart.
• Contained, unverified means the chat ended without a transfer but the outcome is unknown.
• Abandoned before answer means the customer left before receiving a usable result.
• Unresolved means a known failure, repeat request, or explicit statement that the issue remains open.
• Ineligible means the request was outside the pilot’s stated scope or unsafe for automation; it stays in the denominator only if the pilot charter says so.

Do not hide “unknown” inside “resolved.” Unknown is an outcome of the measurement system, not evidence of success or failure. It can be reported separately and used to improve case linkage. If the team excludes unknown episodes from the denominator, show the exclusion rate beside the result. Otherwise, a change in tracking coverage can appear as a performance improvement.

The scorecard: four rates, one case table

Keep a small set of measures that answer different questions. Use a fixed cohort and window, and show the counts behind every percentage. “Eligible episodes” means episodes that meet the pilot’s predeclared scope and have enough data to be classified.

Define the scope before launch: supported topics, customer groups, channels, languages, operating hours, and high-risk cases routed directly to people.

Use this formula for the primary outcome:

verified automated resolution rate = eligible bot-entered issue episodes meeting the task-specific completion rule and showing no qualifying same-issue repeat in the observation window / all eligible bot-entered issue episodes

This is intentionally stricter than “chat ended” and narrower than a general customer-satisfaction score. It counts completed tasks, not just messages, and leaves unknown episodes visible.

If your business cannot reliably link cross-channel contacts, label the metric “observed automated completion” and say that repeat contacts may be missed. Do not call it verified resolution until the evidence supports that name.

Define completion before choosing the bot

Each supported intent needs a completion rule a reviewer can apply without guessing. For an order-status question, the rule might require a current status from the order system, a timestamp or source of that status, and a clear next step if the order is delayed. For a return request, it may require that eligibility was checked and the return instruction or label was actually issued. A message that merely links to a returns page is a response; it is not automatically a completed return request.

Write rules around customer outcomes and system state. “The bot sounded confident” and “the answer was relevant” are not completion conditions. A useful rule may require a confirmed transaction receipt, a successful account update, a customer confirmation, or a follow-up event showing the next step occurred.

Pick evidence proportional to the task. A simple public opening-hours answer does not need the same proof as a refund, a billing change, or an account-security incident. The task rule should identify what evidence is authoritative. If a source-of-truth system reports an action as pending, the bot must not state that it is complete. If data sources conflict, the bot should explain uncertainty or transfer the issue.

A correct escalation can meet the assisted-resolution rule even when automated resolution is zero. This prevents a metric from pressuring the product to overstate outcomes.

Separate intent-level rules from global rules. “Customer asked about an invoice” is too broad: the task may be to find a copy, explain a line item, correct an address, or dispute a charge. Each may have different permissions and evidence. If an intent is too broad to define completion, it is not ready for a resolution KPI; first split it into decision-relevant cases or keep it in a review category.

Link the chat to the issue without over-collecting

Cross-channel measurement requires some way to recognize a repeat issue. The least intrusive reliable join is preferable: a ticket or case identifier already shown to the customer, an authenticated account identifier, or a support-system link created at handoff. Avoid collecting extra personal data just to improve a dashboard. Keep only fields needed for routing, audit, service continuity, and the declared measurement window; restrict access and retention according to the organisation’s data-governance rules.

Matching should combine identity and issue context rather than depend on a fuzzy text similarity score alone. A practical sequence is:

1. use an existing case ID when present;
2. otherwise match a verified account or order reference;
3. then compare a normalized issue category and time window;
4. finally send ambiguous matches to a reviewer.

Never merge two customers because their names look alike. Never mark a repeat as “same issue” solely because the model gives a high semantic similarity score.

At handoff, pass the original transcript or an accessible transcript reference, the customer’s stated goal, the bot’s attempted actions, their tool results, and the reason for escalation. A summary can help, but it should not replace the underlying evidence. Mark which statements came from the customer, which came from the bot, and which came from a business system. A human should be able to see whether “refund submitted” means a successful API response or just a generated sentence.

This data model also reveals a failure that containment hides: the customer may remain in the bot because no human route is visible. Record the transfer control’s availability, the customer’s request for a person, and any blocked or failed transfer. A high containment rate alongside a falling handoff rate is not inherently good; check whether the customer had a real choice and whether unresolved cases increased.

A repeatable pilot protocol for a small team

The following protocol is an original operating template, not a measured benchmark. It is designed to be run with the support system and a spreadsheet before the business invests in a custom analytics pipeline.

First, freeze the pilot charter: purpose, eligible intents, excluded cases, channels, languages, business hours, escalation route, outcome rules, and the length of the observation window. Record the version and start date so later changes do not silently rewrite the baseline.

Second, create a case ledger with one row per issue episode and linked interaction IDs.

Third, decide the matching hierarchy and record its confidence: exact case, authenticated customer plus issue, reviewer-confirmed, or unknown.

Fourth, define the task result event for every included intent.

Fifth, run a pre-launch test set containing ordinary, ambiguous, and failure cases. Do not start with a success metric if the bot cannot reliably produce the evidence the metric requires.

During the pilot, take a reproducible sample of episodes each week. Use a fixed rule, such as every nth eligible episode after sor