IBM is patenting a bias detecting mechanism for chatbots

Computing giant IBM has come up with a system that enables “conversational agents” to automatically detect and correct unintended biases.

The Armonk, New York-based company’s patent application defines conversational agents as dialogue systems that utilize both natural language processing and human language to respond to queries. Implementation of the technology, commonly referred to as chatbots, is expected to widen further, owing to their round-the-clock availability and lower operating costs compared to human agents.

However, the growing use of digital assistants has also exposed some unintended behavior. Aspects of a customer such as race, religion, or livelihood may lead chatbots to generate personalized, but discriminatory responses. To fix this issue, chatbots must be designed with safeguards in place, implemented by programmers that recognize the diverse range of individuals that would likely interact with their creation. UNESCO has recommended solutions to this effect, particularly for combating gender discrimination in AI applications.

IBM is proposing a novel framework for bias detection in conversational agent platforms that makes use of conversational agent definitions, conversational agent chat logs, and user satisfaction statistics. These components help assign scores to digital assistants’ responses, corresponding to levels of unintended bias. Scores are compared between statements with swapped-out terms for customers’ possibly sensitive attributes.

An illustrative embodiment of a term importance scoring mechanism, comparing scores [404, 408] for two utterances [402, 406] with protected attributes [410, 412].

The system would first identify commonly defined protected attribute types using regular expression matching or dictionary matching. Machine learning-based Named Entity Recognition (NER) would find these terms within conversational agent chat logs, and swap sensitive terms with a common equivalent.

Scores are calculated based on user satisfaction rates on chat sessions where protected attributes are detected versus those without. Drops in satisfaction ratings that are greater than or equal to a predetermined threshold are taken to point to unintended bias.

A mitigation recommendation engine may handle responses to detected unintended discrimination. It may suggest additions to existing conversational agents; a chatbot may not already consider “gender” as a protected attribute, for instance. The engine may also recommend retraining of the machine learning model used by the digital assistant. This could involve feeding the algorithm with a new training dataset adjusted for bias potential.

The featured patent application, “Bias Detection in Conversational Agent Platforms”, was filed with the USPTO on January 15, 2020 and published thereafter on July 15, 2021. The listed applicant is International Business Machines Corporation. The listed inventors are Navneet N. Rao, Ming Tan, Haode Qi, Yang Yu, Panos Karagiannis, and Saloni Potdar.