Getting a refund can be the stuff of nightmares—long wait times, hidden fine print exceptions you did not read, or multiple calls to follow-up. But the Bangko Sentral ng Pilipinas’s (BSP)'s new chatbot, “BOB” (BSP Online Buddy), could change this for consumers in the Philippines.

BOB lets consumers file complaints against BSP-supervised financial institutions (BSFIs) through simple chat messages. A report from Smart Parenting tells the story of one consumer named Kitty who got frustrated waiting on a refund from her bank that never came until she learned about BOB and shared her complaint to the chatbot. After waiting months for a response from her bank, Kitty’s account was refunded just six days after reaching out to BOB. 

Kitty’s story shows how new technologies like chatbots can improve consumer protection while reducing the burden on regulators to manually handle every complaint. To understand how BOB works and how it can be improved, Innovations for Poverty Action (IPA) and the BSP analyzed the first year of chatbot data: 1,372,534 messages between BOB and consumers from August 1, 2020, through July 31, 2021.

 

Figure 1: Process for converting BOB chats into consumer protection categories using natural language processing

As Figure 1 shows, IPA and the BSP converted all words from these messages into data points and then sorted the messages into eight sets of keywords related to different consumer protection topics. The chats were analyzed by type of BSFI, demographics of the consumer (when available), and whether the chat was completed or ended in the middle of the conversation. Some of the key findings from our analysis are highlighted below:

 

Figure 2: Volumes of BOB conversations by consumer protection theme

1. The most common complaints were customer service and lending-related issues. Our keyword analysis found that issues with customer service dominated chat and social media complaints. The second most common type of complaint was about lenders (Figure 2). For a regulator like BSP, monitoring volumes across chat topics could provide early signals on which issues consumers are struggling most to resolve with their BSFIs, leading to targeted areas of improvement for the industry while providing insights that can inform regulatory actions.

 

Figure 3: Portion of chatbot interactions that reached the final message

2. High rates of drop-off clusters show where the chatbox user experience could improve. In our sample, only 20% of chats reached the end of their conversation (Figure 3). To better understand these drop-offs, we looked at the most common incomplete last messages to think of ways to improve usage. For example, on the initial BOB menu, users were given three options (new complaint, check complaint status, or FAQ), but our analysis showed many did not understand they should pick only one option and replied with an open text response instead. BOB was directing users who submitted open response answers to use the BSP call center channel, which may discourage new users. One possible fix is to focus on making the initial question clearer and then posing a follow-up question before sending users back to the traditional channel if they do not answer properly the first time.  

3. There is always a need for manual checks to improve classification accuracy and analysis of issues. BOB immediately categorized 55% of the complaints consumers submitted, and when it could not immediately categorize the complaints, the chatbots asked a series of follow-up questions. This led to a correct categorization of 74% of complaints, which, for thousands of messages, is a substantial time-saver for BSP staff who previously had to classify complaints manually. However, as with all machine learning, there is always a need for human review of data points that cannot be easily classified by the algorithm. For any regulator considering a tool like BOB, they must be aware tools like BOB are not a “set it and forget it” solution. They must be prepared to review outliers and train the model as complaints and language evolve.

4. Expansion of SMS usage and promotion of BOB will reach a more diverse coverage of the Philippines. Most conversations in the first year of BOB were from consumers in the National Capital Region (see map), and were either submitted via Facebook or the BSP website. To expand usage in rural areas, promoting the SMS option for using BOB could be explored to help a broader range of Philippine consumers take advantage of the benefits BOB is bringing consumers.

 

Our analysis of the first year of BOB interactions has shown the potential of the chatbot for timely and effective consumer redress, as well as identified areas for improving user interactions and expanding BOB’s reach. BSP is currently updating the scripts of BOB to increase its effectiveness. IPA and BSP will expand the use of natural language processing to turn individual chats into data to monitor consumer experiences and risks in financial services. BOB also holds potential as a tool for the BSP to exercise its market conduct surveillance and consumer protection compliance monitoring powers enshrined in the recently passed Republic Act No. 11765 or the Financial Consumer Protection Act.