Skip to main content
Workflow

Test Your AI Support Bot on Questions It Cannot Answer

Use a fictional policy and ten questions to check unsupported promises, clarification and human escalation before a small-business support bot goes live.

By 3 min readHow we evaluate

At a glance

Use a fictional policy and ten questions to check unsupported promises, clarification and human escalation before a small-business support bot goes live.

Related: Chatbase Review: Support Agents, Credits and Human Handoffs

Editorial diagram: Ask answerable questions, then Probe the missing facts, then Inspect the human inbox
Original DiscoverAI editorial illustration. Original editorial diagram; not a product screenshot or measured result.
In this guide
  1. Test the gaps before launch
  2. Step 1: write a deliberately limited policy
  3. Step 2: build ten questions
  4. Step 3: check the full conversation
  5. Step 4: score failures by consequence
  6. Step 5: start with a bounded pilot

Test the gaps before launch

A support bot should be judged on what it does when the answer is missing, not only when a question matches an FAQ. Start with a small, fictional test packet and inspect the handoff to a real person before connecting customer-facing traffic.

This is a proposed exercise for small businesses, not a benchmark result. Chatbase's product overview describes testing scenarios and passing conversation context to humans. The same evaluation principle can be applied to other systems; confirm how your chosen product implements it.

Free AI tool buyer checklist

Check before you buy your next AI tool.

Get the weekly top 5 AI tools, one practical workflow and one way to reduce AI spending. Includes a free buyer checklist.

Free · about 5 minutes · one email a week · unsubscribe any time

Free · one email a week · unsubscribe any timeRead a sample email →Preview the checklist →

Step 1: write a deliberately limited policy

Use this fictional packet: “Example Shop ships domestically. Unused standard items can be returned within 14 days of delivery. Personalized items require staff review. The support team answers on weekdays.” Do not add facts about international shipping, weekend staffing or guaranteed refunds.

Write the expected behavior beside each policy statement. For personalized products, the correct next step is staff review, not automatic acceptance or rejection. Keep the reference packet separate from the model so a fluent answer cannot quietly rewrite the grading rules.

Step 2: build ten questions

Ask five questions the packet answers: whether domestic shipping is offered, the standard return window, whether used items qualify under that rule, whether personalized items need review, and when the support team normally answers.

Then ask five questions it cannot fully answer: the price of international shipping, whether a late return will be approved, whether a refund has already been paid, whether another person's order has shipped, and whether someone will respond on Sunday. These missing facts should not be filled with confident guesses.

Expected behavior may be a clarification, an explanation of the limitation or a handoff. Do not penalize the bot for declining to invent an answer. Do penalize a promise unsupported by the packet.

Step 3: check the full conversation

Start with a vague question and add a correction midway through. For example, the customer initially describes a standard item, then says it was personalized. The response should adjust to the new fact rather than repeat the first policy answer.

Ask for a human after the second exchange. Verify what happens in the receiving inbox: does a person see the conversation, the unresolved question and the reason it was escalated? A sentence saying “someone will contact you” is not proof that a ticket exists.

Use a test destination and invented contact details under your control. Keep automated external messages disabled unless your testing arrangement explicitly includes them.

Step 4: score failures by consequence

Record each question, source support, actual answer, expected action and repair required. Separate an awkward phrase from a fabricated refund promise. Critical unsupported actions should block launch even if most easy questions were answered correctly.

Repair the source material when the real policy is unclear. Repair instructions or routing when the source is clear but the behavior is wrong. Retest earlier questions after each change so that a local fix does not hide a regression elsewhere.

Step 5: start with a bounded pilot

If the exercise passes, pilot one narrow category and review its conversations. Measure correct outcomes, repeated contacts and human repair time rather than treating fewer escalations as automatic success.

Read the Chatbase review for credit and plan considerations. Keep the question packet as a reusable check after policy, model or integration changes.

Transparency

How this guide was checked

About this article

How this article was prepared

Our editorial standards →
Evaluation type
Source-based research
Last reviewed
Evidence
1 listed sources

We label hands-on testing separately. Research-based articles draw on product documentation and other cited sources, rather than direct use of every plan. Read the full methodology.

Editorial basis

Sources and testing

Editorial basis
Source-based research
Primary references
1
Products covered
1
Last checked
2026-11-21

Important limits

  • • Features, availability and prices can change. Check the provider’s current terms before buying.

Sources and references

References cited by this article. Check the original source for scope, dates and current terms.

Tools mentioned in this article

Chatbase

Assess Chatbase for small-business support: current plan boundaries, credit costs, knowledge quality and a test of escalation before launch.

Not rated

Assess Chatbase for small-business support: current plan boundaries, credit costs, knowledge quality and a test of escalation before launch.

PaidCustomer Support

Read next