Why is calibration needed?
AI identifies Topics based on the content of a conversation, not only on keywords. It uses a text instruction that describes which conversations should be included in the Topic and which should be excluded.
The initial description may not cover every real-world scenario. During calibration, the user reviews conversation examples and indicates whether the AI correctly assigned the Topic to the conversation. Explanations of the AI’s errors are used to create an improved version of the definition.
Step 1: Open the Topic
Click the rocket icon in the left-hand menu.
Go to Topics.
Open the Topic whose accuracy you want to check.
Step 2: Create the initial AI definition
If the Topic does not yet have any definition versions:
Click Generate Initial Definition.
Wait for the improved description to be generated.
Review the definition in the Instruction the AI uses section.
Click Continue to Review.
After generation, the system will use the new AI-generated description instead of the manually created description.
Creating the initial definition for a Topic
Step 3: Configure the calibration filter
The calibration filter is optional. Use it when the Topic applies only to a specific set of conversations.
Before applying a filter, you can review the current samples first. Add a filter if they do not match the required Topic.
It is recommended to select the agent or team whose conversations should be reviewed. You can also add a queue, date range, or other available conditions.
For a general Topic, leave All conversations selected. A filter that is too narrow may produce a smaller and less representative sample.
Open Calibration filter.
Configure the required conditions.
Click Apply Filter & Generate New Samples.
Configuring the calibration filter
If you change the filter while unreviewed examples are present, 20 unreviewed samples will be deleted and replaced with a new sample. Previously reviewed examples will be retained.
How to manually add conversations to the review set
If the system cannot automatically find example conversations for the Topic, matching or non-matching conversations can be added manually.
Go to the “Create review set” or “Manage review set” function.
In the upper-right corner, select the option to add matching or non-matching conversations.
After you click the required button, the page containing the full list of conversations will open.
Filter by a relevant tag to narrow the search for matching conversations.
When you find the required conversation, select it using the checkbox on the left.
The selected conversations will be added as matching or non-matching, depending on the option selected earlier.
Step 4: Evaluate how the AI selected the Topic for the conversation
The review interface contains:
a link to the full conversation, where you can read the complete transcript or listen to the conversation;
the highlighted excerpt used by the AI for evaluation;
an explanation of the decision and the confidence level as a percentage;
the Yes, No, and Unclear response buttons.
The option recommended by the AI is highlighted in red, green, or yellow. If you agree with the AI’s decision, click the highlighted option. If you disagree, select the correct answer.
Conversation review, the AI decision, and available response options
Step 5: Explain an incorrect decision
If your answer differs from the AI’s decision, the “Explain your label” window will open.
Enter a specific explanation of your decision.
Specify the excerpts or circumstances from the conversation that support your evaluation.
Click Save.
The AI uses this explanation to create a new definition, so it must be precise.
Field for explaining your evaluation
Step 6: Complete the sample review
Repeat the evaluation of the selected Topics for all suggested conversations. When complete, the system will report that all calibration examples have been reviewed and will begin preparing a new Topic definition.
A new review sample can be created after the definition has been prepared.
Creating a new Topic definition may take more than an hour. During this time, the system performs semantic analysis and recalibration.
Step 7: Review the result
After processing is complete, the system will display the updated accuracy and improved instruction in the Instruction the AI uses field.
If the result is not satisfactory:
Click Create Review Batch and Review More Conversations.
Review another 20 conversations.
The additional sample gives the AI more training examples and enables it to create a more precise definition.
Updated accuracy and creation of an additional review sample
Step 8: Activate the required version
Open the Versions panel on the right-hand side of the screen.
Compare the available definition versions.
Select the required version.
Activate the version by clicking the checkmark icon next to it.
Selecting a Topic definition version
After the selected version is activated, the accuracy score will be displayed in the main Topics list.
Calibration is now complete: the activated version is used to categorize conversations from this point onward.
How to evaluate Topic accuracy
After you review the conversation examples, the platform displays the Topic accuracy score. It helps you understand how accurately the system identifies conversations that belong or do not belong to this Topic.
The following elements are used to configure a Topic:
the initial Topic description — a human-written description of its content and purpose;
Include when — the conditions under which a conversation should be assigned to the Topic;
Exclude when — the conditions under which a conversation should not be assigned to the Topic.
The next steps depend on the resulting accuracy score:
Below 70% — low accuracy. In the “All Conversations” section, find conversations whose context may match this Topic and read them to understand the context more fully. Manually assign the conversations you found as examples for the Topic. Then change the initial human-written Topic description so that it more accurately reflects the content of these conversations.
From 70% to 79% — accuracy needs improvement. In the “All Conversations” section, find conversations whose context may match this Topic and read them to understand the context more fully. Then review the Include when and Exclude when sections. Determine which conditions should be added, refined, or removed so that the system can distinguish matching and non-matching conversations more accurately.
From 80% to 89% — the Topic is almost ready. Carefully review the initial description, Include when, and Exclude when. Determine which nuance is missing for more accurate conversation identification.
90% or higher — satisfactory accuracy. This result is considered sufficient for using the Topic.