Test the AI before customers do
Write the questions that matter, say what a good answer does, and let Ha Tang grade the assistant on them after every change.
Updated on
The Tests tab checks the assistant against questions your customers really ask. You write the question and what a good reply does; Ha Tang asks the assistant three times and grades each answer from 0 to 100. Run the tests after you change the knowledge, the instructions or a situation, and you will know whether you made things better or worse.
Add a test
- Go to Customer Support → AI assistant → Tests.
- Under Add a test, fill in:
- Name: a short label, for example "Shipping fee to Da Nang".
- Channel and Office hours: the context to pretend, for example Zalo OA and Outside office hours.
- Earlier chat (optional): previous messages, one per line, starting with "Customer:" or "AI:".
- Customer says: the message to answer.
- Expected: what a good reply does, in plain words. "States the shipping fee and time to Da Nang; promises no exact date" or "Apologizes and hands off to a person".
- Click Add.
Run the tests
Click Run all at the top, or Run under one test. Tests show Queued, then Grading; reload the page after a few minutes to see the results.
For each test you get:
- the score, the average of the three grades, in green from 80, orange from 50, red below;
- the spread, the gap between the best and the worst of the three answers. A spread of more than 30 points means the assistant is not consistent yet on this question;
- See 3 answers: each reply with its grade and a one-sentence reason from the grader.
The page also shows the average score over all tests.
The assistant answers tests exactly as it would answer a customer: same knowledge, instructions, situations, products and tools. A hand-off counts as right only if the expectation asks for one, or the question really needs a person.
Write good tests
- Start from real questions, from Unanswered questions and from conversations agents had to correct.
- One thing per test. "Gives the right fee" is easy to grade; "gives the fee, the time, the return policy and a discount code" is not.
- Test the hand-offs too: refunds, complaints, a customer asking for a person. The expectation is "hands off to a person, without promising a refund".
- Test what must not happen: "does not invent a delivery date", "does not offer a discount".
What to do with a low score
| What you see | What it usually means | What to do |
|---|---|---|
| Low score, small spread | The knowledge is missing or wrong | Fix the source, or answer the question in Unanswered questions |
| Good average, large spread | The knowledge is vague or contradicts itself | Make the answer one clear Q/A pair; remove the old version |
| It hands off when it should answer | The passage is not found | Use the customer's words in the question; check Knowledge used in Chat with the AI |
| It answers when it should hand off | Your instructions or a situation push it to keep going | Add a situation with Hand off to an agent |
Useful next steps
Was this article helpful?
Related articles
- Answer the questions the AI could notTurn the questions the assistant handed off into new knowledge, and correct a reply straight from the inbox.
- Teach the AI how to handle situationsRules for specific cases, such as wholesale enquiries, angry customers or after-hours chats, that say what the AI says and what happens next.
- Check what the AI did and what it costThe AI runs log shows every answer, draft, hand-off and test the assistant made in the last 30 days, with its outcome and cost.
Not finding what you are looking for?
Our team answers in the chat, in English or Vietnamese.