Have you conversed with a virtual agent but were unable to get the information you really required? We all have. This often leads to a sub-par customer experience and the need for human intervention. Bridging the gap where the virtual agent learns to cover a wide range of topics pertinent to its end-users is where the need for a feedback mechanism arises.

Conversational AI today is dependent to a great extent on human knowledge. Virtual agents are developed on a model that depends either on rules or intent classification, and the majority of the intelligence comes from training data that is input by a human on the bot-building platform.

“86% of customers believe that there shall be an option to talk to live agents or redirect the live agents’ queries.”
NotifyVisitors

The need for self-learning AI is increasingly prevalent in this context to consistently improve the performance of the bot. Quantiphi’s sQrutinizer is here to solve this problem for you.

The sQrutinizer is a tool that:

Analyzes the conversation logs of your Google Cloud CCAI virtual agent to identify those utterances that have not been understood properly by the agent, which include fallbacks, and false positives.
Provides you with the option to retrain the bot to cover such topics, essentially creating an automated training pipeline.

How it’s done

sQrutinizer analyzes those user utterances or phrases that were not recognized by the Google Cloud CCAI as well as those that had a lower confidence score. Using these fallback user utterances and false positives, sQrutinizer helps developers and conversation designers with these two tasks:

Intent Classification – Classifying user utterances as a possible variation of the training phrases of any of the existing intents.
Intent Generation – Identifying a set of user utterances as a possible candidate for new intent creation.

With sQrutinizer, designers can focus on the most important aspects of designing great conversations, and leave the intent training to sQrutinizer.

The setup is a simple three-step process –

Select the time period for which you want the logs analyzed. E.g., one month
Select either a real-time analysis or a batch processing request
Choose to retrain the bot with specifically suggested intents provided by sQrutinizer

You can also choose between multiple clustering and classification models so as to achieve the maximum accuracy possible.

How it works

A user initiates a chat with a virtual agent (VA)
The VA is unable to answer the user’s queries, so the call is transferred to a live agent
Human-agent handles further queries
sQrutinizer: In the backend, the conversational logs are fed to sQrutinizer to analyze why the virtual agent failed to answer or why a handover happened, it could be any of the following or more.
1. The user asked something out of bot knowledge
2. The user utterance was not part of the training dataset
3. A step in the conversational flow failed to be recognized

sQrutinizer helps ensure that the virtual agent itself will learn to take into account such queries that it was unable to answer earlier, thereby expanding the virtual agent knowledge base via retraining the NLU.

Impact on VAs

The three primary benefits of the feedback loop are:

VA accuracy is improved significantly
The need for human intervention is reduced to a minimum
The overall conversational experience for end-users is enhanced

To conclude, sQrutinizer helps conversation designers and developers monitor the fallback and false-positive utterances of their virtual agents. Using the results from sQrutinizer, developers can either update the existing intent training to improve the virtual agent performance and efficiency or create new intents to broaden the knowledge base of the bot.

Get in touch with our experts today to explore sQrutinizer and drastically improve your bot performance and, in turn, your customers’ experience.

Contributed by: Kanishk Mehta and Krish Kalro