Finding the right metrics for evaluating your bot

Here’s the good news for companies building AI-powered virtual agents (aka bots) for customer service: Research from Accenture suggests that 80 percent of support chats or calls can be resolved by bots that have “good design.” The bad news is, the definition of good design changes depending on your business case.

Historically, many bot builders across industries have considered natural language understanding (NLU) a silver bullet for enabling extended, open-ended conversations. Indeed, too many implementers of failed bots have confessed to me that the plan all along was to support a handful of intents and have the bot learn phrasing and vocabulary variations “on the fly.” Unfortunately, that approach too often results in failure because NLU technology, while impressive and advancing rapidly, is not yet up to the task of artificial general intelligence.