In this guide
Test the gaps before launch
A support bot should be judged on what it does when the answer is missing, not only when a question matches an FAQ. Start with a small, fictional test packet and inspect the handoff to a real person before connecting customer-facing traffic.
This is a proposed exercise for small businesses, not a benchmark result. Chatbase's product overview describes testing scenarios and passing conversation context to humans. The same evaluation principle can be applied to other systems; confirm how your chosen product implements it.
Free AI tool buyer checklist
Check before you buy your next AI tool.
Get the weekly top 5 AI tools, one practical workflow and one way to reduce AI spending. Includes a free buyer checklist.
Step 1: write a deliberately limited policy
Use this fictional packet: “Example Shop ships domestically. Unused standard items can be returned within 14 days of delivery. Personalized items require staff review. The support team answers on weekdays.” Do not add facts about international shipping, weekend staffing or guaranteed refunds.
Write the expected behavior beside each policy statement. For personalized products, the correct next step is staff review, not automatic acceptance or rejection. Keep the reference packet separate from the model so a fluent answer cannot quietly rewrite the grading rules.
Step 2: build ten questions
Ask five questions the packet answers: whether domestic shipping is offered, the standard return window, whether used items qualify under that rule, whether personalized items need review, and when the support team normally answers.
Then ask five questions it cannot fully answer: the price of international shipping, whether a late return will be approved, whether a refund has already been paid, whether another person's order has shipped, and whether someone will respond on Sunday. These missing facts should not be filled with confident guesses.
Expected behavior may be a clarification, an explanation of the limitation or a handoff. Do not penalize the bot for declining to invent an answer. Do penalize a promise unsupported by the packet.
Step 3: check the full conversation
Start with a vague question and add a correction midway through. For example, the customer initially describes a standard item, then says it was personalized. The response should adjust to the new fact rather than repeat the first policy answer.
Ask for a human after the second exchange. Verify what happens in the receiving inbox: does a person see the conversation, the unresolved question and the reason it was escalated? A sentence saying “someone will contact you” is not proof that a ticket exists.
Use a test destination and invented contact details under your control. Keep automated external messages disabled unless your testing arrangement explicitly includes them.
Step 4: score failures by consequence
Record each question, source support, actual answer, expected action and repair required. Separate an awkward phrase from a fabricated refund promise. Critical unsupported actions should block launch even if most easy questions were answered correctly.
Repair the source material when the real policy is unclear. Repair instructions or routing when the source is clear but the behavior is wrong. Retest earlier questions after each change so that a local fix does not hide a regression elsewhere.
Step 5: start with a bounded pilot
If the exercise passes, pilot one narrow category and review its conversations. Measure correct outcomes, repeated contacts and human repair time rather than treating fewer escalations as automatic success.
Read the Chatbase review for credit and plan considerations. Keep the question packet as a reusable check after policy, model or integration changes.
Transparency
How this guide was checked
About this article
How this article was prepared
- Evaluation type
- Source-based research
- Last reviewed
- Evidence
- 1 listed sources
We label hands-on testing separately. Research-based articles draw on product documentation and other cited sources, rather than direct use of every plan. Read the full methodology.
Editorial basis
Sources and testing
- Editorial basis
- Source-based research
- Primary references
- 1
- Products covered
- 1
- Last checked
- 2026-11-21
Important limits
- • Features, availability and prices can change. Check the provider’s current terms before buying.
Sources and references
References cited by this article. Check the original source for scope, dates and current terms.
Tools mentioned in this article
Chatbase
Assess Chatbase for small-business support: current plan boundaries, credit costs, knowledge quality and a test of escalation before launch.
Assess Chatbase for small-business support: current plan boundaries, credit costs, knowledge quality and a test of escalation before launch.
Read next