Why This is Needed and How It Works
How it works inside: The AI assigns topics to conversations based on semantic meaning rather than just keywords. It relies on a specific text definition ("Instruction the AI uses") that tells the model what defines the topic and what scenarios to exclude.
Why check accuracy (calibration): Your initial description might not perfectly capture the real-life nuances of your customer calls. When you evaluate sampled conversations, you provide feedback on whether the AI labeled the call correctly. Based on your explanations for any incorrect decisions, the AI automatically formulates a new, improved topic description. This iterative process allows you to measure and increase the topic's accuracy score (e.g., pushing it to 90% or higher).
Step 1: Navigate to the Topics Module
Click on the Rocket icon in the left sidebar.
Go to the Topics section.
Click on and open the specific topic for which you want to check the accuracy.
Step 2: Generate the Initial AI Definition
Inside the topic, you will see the description that you created earlier.
If no definition versions have been created yet, click the purple button labeled GENERATE INITIAL DEFINITION.
Generating the AI definition of the topic requires some time. Once it finishes, an improved AI description will appear. This means topics will now be assigned to conversations based on the AI-generated description rather than the one you created manually before.
Click CONTINUE TO REVIEW to proceed to the accuracy check.
Step 3: Apply Filters for Calibration
To ensure the AI learns from relevant data, you should filter the sample calls used for review:
Click on the Calibration filter (optional) dropdown menu.
We recommend using the Agent / Team filter to select the specific team whose calls need this topic tracked. You can also apply additional filters, such as specific queues or date ranges, if needed. Important: Use a filter when the topic only applies to a specific part of your traffic, so the review and learning stay focused on relevant calls. Leave it set to "All conversations" only when the topic applies broadly—narrowing a broad topic gives fewer, less representative samples and can skew the measured accuracy.
Click APPLY FILTER & GENERATE NEW SAMPLES.
Step 4: Evaluate Conversations (The Review Interface)
Now the accuracy check begins, and you will see the first conversation evaluated by you and the AI together.
The interface contains four main evaluation elements:
1. Conversation Link: You can follow the link to read the full transcript or listen to the conversation if you need more context.
2. Highlighted Transcript Segment: This shows the specific highlighted phrase in the dialogue that the AI took as a base for its evaluation. Read this segment carefully.
3. Rationale and Confidence Score: This displays an explanation of the AI's decision along with a percentage level showing how sure the AI is about its prediction.
4. Decision Buttons: You will see the AI's suggestion highlighted in color (red, green, or yellow) indicating whether the topic is related to the conversation.
Click the color-highlighted button if you agree with the AI's decision.
If you do not agree, choose the button that suits the conversation best (YES, NO, or UNCLEAR).
Step 5: Train the AI on Mistakes (When Choosing "NO")
If you disagree with the AI's decision, the system will ask you to provide an explanation in a pop-up window:
In the Explain your label box, enter the specific evidence or reasoning that supports why the label should be different. Because the AI will formulate new descriptions based directly on this explanation, be as precise and accurate as possible.
Click SAVE.
Step 6: Complete the Review Iteration
Repeat this exact evaluation process for all sampled conversations.
Once you finish, you will see a confirmation message stating: "All pending calibration samples have been reviewed". The system will indicate that the topic definition is being prepared, and new review samples can be created after it finishes.
Important Note on Processing Time: Generating the new AI definition may take over an hour. There is no need to panic if loading takes a while. Behind the scenes, the system is conducting a deep semantic analysis of your reviewed calls and carefully recalibrating its instructions. We are continuously working to optimize processing speed while ensuring maximum accuracy.
Step 7: Analyze Results and Push for Higher Accuracy
Once processing is complete, you will see the result of your accuracy check (for example, Accuracy improved to 90.9%) along with an improved new description of the topic.
Under the Instruction the AI uses section, you can read the updated text definition that the AI actually follows when categorizing calls.
If you are not satisfied with the accuracy result, click the purple CREATE REVIEW BATCH and REVIEW MORE CONVERSATIONS button. Reviewing 20 more conversations provides more labeled examples, giving the AI a bigger database to learn from and making the evaluation more precise.
Step 8: Activate the Final Version
When you are finished checking the topic's accuracy and are satisfied with the performance:
Look at the Versions panel on the right side of the screen.
Choose the best version of the topic description.
Activate it in the system by clicking the tick icon next to that specific version.
Finally, after reviewing and approving the best version, you’ll be able to see the accuracy of the topics throughout the entire list.
Congratulations — you’ve learned how to review topics on the Ender Turing platform. Good luck!