In past few months, we partnered with Speed Reply on developing a smart customer reply solution. SpeedReply is a smart solution that identifies intent of comments on social media and then generate up to three replies for the related comment. Users can further customised the selected comment and submit replies through our system. Our goal is to help brand become more responsive and reduce the workload of social media page managers.

SpeedReply Smart CRM Solutions

Here is the demo of our product. Enjoy!

Current Business Problem

As a consumer, we all have poor serivce experiences from companies and service providers. Nowadays, as social media becomes universal to anyone, most customers or even potential customers tend to express their opinions on companies’ social media page.

These comments can be a double-edged sword when it comes to brand image. Positive reviews can improve brand image while negative comments will do the opposite. Hence, brand’s responsiveness and quality of replies are essential to close the service gap.

According to a recent social media trends study from Global Web Index(GWI), more than 20% of respondents indicating that number of social media likes and positive comments might potentially enhance their chance of purchase. Besides, over 40% of respondents say that they will use social networks to do research on products.

Our Speed Reply solution can help brands improve customers’ perception and build great brand image.

Current Products on the Market

The current CRM solution on the market can only offer sentiment analysis in terms of product functions. However, SpeedReply is even several steps further. We can offer sentiment / intent analysis together with suggested reply as an all-in-one solution.

Given the fact that machine learning tools focusing on Cantonese are virtually non-existent, these existing products mainly support English language and do not support Cantonese. Therefore, we believe that our product can help address the business needs of handling Cantonese comments.

SpeedReply Development Approach

Our development approach will be explained in detail through the below steps:

1. Data Collection
2. Data Pre-processing
3. Model Building
   • Intent Prediction Model, Negative Reply Recommender Model, Positive Chitchat Model
4. Deployment

Step 1. Data Collection

Our approach is to train our model with Natural Language Processing(NLP). Based on our observation, we noticed that there are two major types of replies.

• Professional and sincere tone for negative comments
• Light-mannered and friendly tone for positive comments

As a result, we collected data from two different sources in order to overcome the above obstacles. We created two datasets for different models.

• A well-known online shopping platform that gives professional and official replies towards negative comments.
  • This dataset is used for Intent Prediction Model, Negative Reply Recommender Model.
  • We extracted comments, replies, and number of stars given by users to create our dataset. For the star rating system on the platform, score 1 out of 5 is the lowest.
• A major airline with active social media presence and frequent interaction with users on positive comments.
  • We extracted comments and replies for Chitchat Model.

Step 2. Data Pre-processing

2a. Dataset for Intent Prediction Model, Negative Reply Recommender Model

i. Stoplist

After collecting data, we removed punctuations to prevent them from confusing the classification model. Also, since stopwords are meaningless to the sentence, they are removed using our customised stoplist. As the current libraries are only available in English or Mandarin, we have to customise one ourselves. Here is our stoplist example.

ii. Token List

Apart from stoplist, we also had to customize a tokenisation list in order to cut sentence in Cantonese. Here is our token list example.

iii. Label Intent

In our classification model, we will create four intents (i.e. positive, refund, enquiry and feedback). Therefore, we need four labels for our data.

Besides, since we used supervised learning approach to train the intent prediction model, we labelled the comment sentiment based on number of stars. Comments with one to two stars were labelled as negative while the rest were positive. For the negative comments, we further classified them into enquiry, refund and feedback.

In order to label intent in a more accurate way, we used a word2vec model to find as many synonymous words for each intent as possible.

iv. Handle Imbalanced Data

Since the data are collected in real-life, it is common that the data are imbalanced. We used SMOTE resampling techniques to oversample the data. So that a more accurate result can be obtained when predicting the intent for each comment.

v. Eliminate Irrelevant Data

Since this dataset is used to train two models, the data required in each models are different. For Intent Prediction Model, we will use all comments and their own reply model to train data. For Negative Reply Recommender Model, we will remove the comments without replies in order to teach the model to generate replies. Also, all specific data like order number, address, product name were removed from the replies.

2b. Dataset for Chitchat Model

Since our chitchat model is mainly used to interact with positive comments, the negative comments in this dataset were removed. As we hope to generate some general replies for users to further customise on, the customer details like order number, customer name in each replies were also removed. The comments without replies are also removed.

Step 3. Model Creation

As mentioned above, we have built three models for our SpeedReply solution. They are Intent Prediction Model, Negative Reply Recommender Model, Positive Chitchat Model. Generally speaking, our SpeedReply solution works by predicting the intent for incoming comments. If the predicted intent is positive, the system will generate response in chitchat tone using Positive Chitchat Model. For the remaining intent, Negative Recommender Model will be used. Both models will generate up to three responses for user to choose from. After further customisation, the related replies will be submitted to social media platform. The below flowchart illustrates the general workflow of our SpeedReply solution.

3a. Intent Prediction Model

We used LSTM model to predict intent. The accuracy of this model is around 78%.

3b. Serious Tone: Negative Reply Recommender Model

Nowadays, customers always approach staff through e-channels with various issues, like enquiries, product/ service reviews, complaints, after-sale services. However, it is very time-consuming to reply to each comment one by one. To ease the workload of staff, we created a Doc2Vec model for generating suggested replies. Since each company will have different templates for staff to follow and further customize on, our model will suggest related replies based on the customer comments.

Before training the model, we tokenized and removed the stopwords from the data. And then tag each comment as one document. After training, we used Doc2Vec Model to find the top 3 most similar comments from the dataset and get the related replies as recommendations. As the replies in the dataset have already been customised based on the official templates, it is believed that the time for writing unique replies can be drastically reduced. All they have to do is to add back some specific details like order number and product name.

The accuracy of this model is 70%.

3c. Light Tone: Positive Chitchat Model

Maintaining customer relationship is one of the essential ways to build up good brand image. Therefore, not only should we handle negative comments, but also we should respond to those positive comments to enhance customer engagement.

To achieve this, we created a Chitchat Model to automatically generate some replies based on those comments. In this model, we used a pre-trained GPT2 Model as base model for training. We have to pair up the comments and replies and then add back the classifier and separator before training. The example in English will be [CLS] Really love your service![SEP] Thank you! [SEP].

Since we are using bidirectional approach for training, we build up 2 GPT2 Models: one for forward training, one for backward training. So that we can find the best generated response.

For response prediction, like the traditional approach, the model generates the response word by word. To enhance the accuracy, the model will try using the generated response to backward predict the comment to calculate the accuracy. The model will generate up to three alternative responses for users to choose from. And then they can select and further customize the reply based on the generated results. It is believed that the work efficiency can be enhanced and hence the brand can be more responsive when it comes to customer relationship management.

The accuracy of this model is 60% which is pretty good for this kind of model. In long term, reinforcement training is required in order to enhance the accuracy and diversity of generated replies.

Step 4. Model Deployment

We created a web app for our solution using Django. In our web app, user can mark as read.Foreseeing the potential to add more components/features to Speed Reply in the near future, our final data are stored on google cloud SQL, which gives flexibility for us to visualise data or create dashboard to monitor social media engagement performance in the future.

Our app has been successfully deployed to Gcloud. All user has to do is to login our system. Whenever there is a new comment from Facebook, our system will capture such data, preprocess the data, predict intent, perform machine learning and show with up to 3 best recommended reply suggestions to the user. The user then has a choice to click on the suggested reply, click send, and the reply will seamlessly be sent to the particular comment.

Challenges

1. Sample Size

Sample size was a main concern. Although there are many online shops and social media touch points, many of them does not offer comments section. Even for those who do, most of them do not reply (Which highlights the importance of why Speed Reply can step in and improve customer journey).

Although we could have increased the data source by extracting the vast amount of data from Taobao and t-mall, we did not proceed with this approach. This is because the reply style and tone from Mainland China are vastly different from Hong Kong. For example (Per below), stores will reply with compliments that will be deemed as exaggeration via Hong Kong users. As we are predominantly building a tool for Hong Kong businesses replying to Cantonese speaking users, we were better off using datasets from Hong Kong websites and social media pages.

2. Named Entity Recognition

To provide an instant understanding beside speed reply, we also applied NER(Name Entity Recognition). NER can extract and label keyword entity with color from the comment, such as product name, detail of request, feeling.

To achieve this, we label all the keyword based on the format of NER (‘B-XXX’,’I-XXX’,’O’) and make use of Bert to train the data. After that, the entity can be clearly recognized and visualized from every customer input.

Yet, due to the model size, it takes time to load NER every time. Considering the efficiency of reply recommendation, we chose not to deploy NER to our demo.

Conclusions and Future Extensions

In a foreseeable future, more and more customers will continue to flock to the internet to voice their opinions. As a result, continuous improvement on response rate is essential in order to improve their customer’s journey.

For now, Speed Reply can offer suggestions in very generalized manner; with increased dataset, stopwords, tokenization list and optimizations on machine learning model (Such as neural network parameters and layers) - more precise, customised response recommendation can be achieved and further provide positive customer journey to the customers.

Furthermore, Speed Reply can only process text comments/replies. However, as customers now also have the choice of using multimedia to reply (Such as animated GIFs as per below), we can look into implementing computer vision to process image related comments.

Aside from reply suggestions, we will also look to assist brands in other aspects of online presence such as:

1. Content Generator - Content (In terms of Facebook post, website articles, etc) auto-generator based on Natural Language Generator (NLG) models.
2. Chatbot - RasaX-powered chat system to handle task related customer needs (Such as refunds).
3. Dashboard - Social Media Engagement KPIs

Q&A Section

Question:

Why don’t you create a fully-automated solution?

SpeedReply:

As we understand that all brands value heavily on customer relationship, customer comments might be overlooked or omitted, which means the brand might dissatisfy customer or miss some important feedbacks if we fully automate our solution.

References

1. Gmail Smart Compose: Real-Time Assisted Writing: https://research.google/pubs/pub48231/
2. Smart Reply: Automated Response Suggestion for Email: https://research.google/pubs/pub45189/
3. Uber’s Smart Reply System: https://eng.uber.com/one-click-chat/
4. Deloitte Conversational AI: https://www2.deloitte.com/content/dam/Deloitte/au/Documents/strategy/au-deloitte-conversational-ai.pdf
5. Sentiment Augmented Attention Network for Cantonese Restaurant Review Analysis https://sentic.net/wisdom2019xiang.pdf

Remarks

As this project is a partnership with SpeedReply, the codes will not be disclosed in detail. Sorry for any inconvenience caused.

Link to SpeedReply Official Website