Predicting Amazon customer reviews with deep confidence using deep learning and conformal prediction

Abstract

In this investigation, we have shown that the combination of deep learning, including natural language processing, and conformal prediction results in highly predictive and efficient temporal test set sentiment estimates for 12 categories of Amazon product reviews using either in-category predictions, i.e. the model and the test set are from the same review category or cross-category predictions, i.e. using a model of another review category for predicting the test set. The similar results from in- and cross-category predictions indicate high degree of generalizability across product review categories. The investigation also shows that the combination of deep learning and conformal prediction gracefully handles class imbalances without explicit class balancing measures.

Keywords:

• Amazon customer reviews
• machine learning
• conformal prediction
• deep learning
• natural language processing
• temporal test sets

1. Introduction

Sentiment analysis investigates consumers’ opinion, or sentiment, toward certain products or entities and is one of the fastest growing research areas in computer science. For recent papers and reviews, see, for instance, (Liu, 2010; Mäntylä et al., 2018; Pang & Lee, 2008; Redhu et al., 2018; Yaakub et al., 2019). Sentiment analysis aims to automatically determine the expressive direction (sentiment) of user product reviews (Luo et al., 2016) and the demand for such analysis has increased due to growing requirements of analyzing and structuring information in various social media in the form of unstructured data (Haenlein & Kaplan, 2010). This, in turn, has been prompted by the growing power of user-generated product reviews influencing consumer’s purchasing decisions known as electronic word-of-mouth (eWOM). eWOM is defined as “the dynamic and ongoing information exchange process between potential, actual, or former consumers regarding a product, service, brand or company, which is available to a multitude of people and institutions via the Internet” (Ismagilova et al., 2017) and has become an important source of information for consumers. One way to perform this analysis, that has proven to be successful, is through the use of machine learning.

Machine learning has, for the past decade, come to play an increasingly more important part in sentiment analysis (Ding et al., 2020; Hossain et al., 2013; Yaakub et al., 2019). The recent advent and progress in deep learning methodology (Goodfellow et al., 2016; Wang et al., 2020) has further increased the usage and utility of machine learning in the sentiment analysis domain (Ain et al., 2017; Araque et al., 2017, 2018).

A problem in machine learning is many times the lack of rigidly defined confidence measures for predictions resulting from such approaches (Bhattacharyya, 2013; Dinga et al., 2019) and, thus, the inability to answer the question: from which part of review (feature) space can reliable predictions be expected?

Conformal prediction is a framework, within the family of confidence predictors, that enables such rigidly defined confidence measures on an instance level (i.e. for each review) given that data are exchangeable (Vovk et al., 2005). It should here be noted that all machine learning methods in general assume at least exchangability or, many times, the stricter assumption of the data to be IID (Independent and Identically Distributed) which means that there are no new requirements introduced by conformal prediction when building confidence predictors.

Deep learning has previously been used for sentiment analysis of Amazon data using word vectors and a recurrent neural network of almost 5.2 million reviews from the categories: beauty, book, electronic, and home (Shrestha & Nasoz, 2019) also in combination with density-based conformal prediction (Messoudi et al., 2020) where one of the three investigated datasets was the well-balanced (1:1) 50k IMDb review dataset (Maas et al., 2011).

In this work, we analyze Amazon customer reviews using deep learning and Mondrian conformal prediction (Vovk et al., 2005), i.e. class-wise conformal prediction, related to 12 Amazon product categories (Table 1), with highly imbalanced sentiment classes.

Table 1. Dataset characteristics.

	Training set^a						Test set
Category	Successful^b	Class neutral and positive number of reviews	Class neg number of reviews	Percentage of class negative reviews	Class negative median words/review	Class neutraland positive median words/review	Class neutral and positive number of reviews	Class negative number of reviews	Percentage of class negative reviews	Class negative median words/review	Class neutral and positive median words/review
movies_and_tv	yes	1,490,904 (799,490)	206,629 (91,934)	12.2 (10.3)	115 (76)	88 (48)	1,487,689	162,793	9.9	14	10
cds_and_vinyl	yes	954,874 (375,101)	76,341 (19,648)	7.4 (5.0)	105 (76)	99 (44)	393,054	13,294	3.3	36	13
books	yes	11,656,232 (9,191,149)	1,001,745 (692,288)	7.9 (7.0)	88 (69)	62 (49)	13,601,159	828,643	5.7	46	36
electronics	yes	2,268,955 (1,906,881)	311,990 (248,841)	12.1 (11.5)	75 (67)	52 (46)	3,629,964	455,278	11.1	42	22
office_products	yes	202,853 (185,812)	18,947 (16,190)	8.5 (8.0)	70 (63)	42 (39)	507,459	34,864	6.4	39	17
cell_phones_ and_accessories	yes	276,950 (269,914)	41,077 (39,370)	12.9 (12.7)	45 (44)	37 (36)	709,108	97,290	12.1	32	21
musical_ instruments	No	72,921 (68,760)	4,784 (4,486)	6.2 (6.1)	70 (68)	48 (46)	138,020	9578	6.5	42	21
patio_lawn_and_ garden	No	176,588 (167,036)	22,680 (20,817)	11.4 (11.1)	59 (56)	43 (41)	509,943	58,541	10.3	33	19
sports_and_ outdoors	No	694,794 (667,155)	57,191 (54,142)	7.6 (7.5)	57 (55)	43 (42)	1,846,245	149,287	7.5	36	22
grocery_and_ gourmet_food	No	312,867 (285,490)	29,125 (26,034)	8.5 (8.4)	55 (53)	40 (38)	695,311	59,874	7.9	31	17
arts_crafts_and_sewing	No	109,150 (107,346)	6,265 (6,118)	5.4 (5.4)	50 (49)	32 (31)	323,548	18,922	5.5	33	15
clothing_shoes_and_ jewelry	No	2,119,165 (2,069,843)	225,723 (220,110)	9.6 (9.6)	41 (41)	34 (34)	7,810,225	873,005	10.1	27	19

^a Values in parenthesis are for period 2011–2014 only.

^b Deep learning/Conformal Prediction model successfully built (yes) or not (no).

2. Method

2.1. Data collection

The data used in this investigation are a set of 12 categories of five-core product reviews collected from amazon.com (Table 1) (https://nijianmo.github.io/amazon/index.html, accessed 23 April 2020). The five-grade (1–5) grading system was converted into a binary negative (1–2) (class negative) vs. neutral-positive (3–5) (class neutral and positive) classification system. All datasets are highly imbalanced (Table 1) with imbalance ratios of between ∼6.7:1 and 17.4:1.

2.2. Data processing

The Amazon reviews consist of several sections where the following sections were used in this analysis:

“overall” =  sentiment, “reviewTime,” “reviewerID,” “asin,” “reviewText,” and “summary”:

All non-alpha numeric characters were removed from the “reviewText” section and the “summary” section was then added to the “reviewText” section.

The “reviewText” section of each dataset was split in a temporal manner into a training set consisting of reviews before 2015 and a test set consisting of reviews after 2014. For multiple reviews made by the same reviewer (reviewerID) and item (asin), only the oldest review was kept regardless of whether the review was published before 2015 or after 2014.

The training set was subsequently randomly split into a proper training set (80%) and a calibration set (20%).

English stop words were imported from the Natural Language Toolkit (nltk) corpus and removed from the “reviewText” section. The vocabulary size for the tokenizer was set to 20,000 words and the tokens were generated from the proper training set and later transformed into a sequence of integers for each review and, if needed, 0-filled (right padded, i.e. adding zeros at the end of the sequence) to a constant length of 250.

The data processing pipeline is depicted in Figure 1.

Figure 1. Data processing pipeline.

2.3. Deep learning and conformal prediction

2.3.1. Deep learning architecture

The deep neural network (DNN) layout used in this study was inspired by the Knime example on sentiment analysis (https://hub.knime.com/knime/spaces/Examples/latest/04_Analytics/14_Deep_Learning/02_Keras/08_Sentiment_Analysis_with_Deep_Learning_KNIME_nodes, accessed 10 April 2020) and the code at https://github.com/keras-team/keras/blob/master/examples/imdb_lstm.py (accessed 10 April 2020). The layout consists of three parts – an embedding layer converting each review into a fixed length vector), the long short-term memory (LSTM) part enabling the network to learn from past observations, and an output layer providing the probability of a review being negative or neutral and positive. The layout is depicted in Figure 2 (DNN model). Check-pointing, in this case saving the best model according to the validation set loss function, was also used in order not to over-train the model.

Figure 2. Model building using DNN and Conformal Prediction.

See Supplementary material for the DNN model building code.

2.3.2. Mondrian conformal prediction

Conformal prediction (CP) performs a recalibration of the original output from the DNN model in order to account for class imbalances and Mondrian conformal prediction (MCP) performs this recalibration separately for each class (negative and neutral and positive, respectively).

Thus, in the MCP part of the model building process, the output from the DNN model (probabilities for each of the two classes) for each review in the test was compared to the corresponding list, sorted in descending order, of calibration set probabilities for each class. The fact that a separate list is used for each class constitutes the Mondrian part of CP. The user has to set a significance level (the percentage of prediction errors that is acceptable) and the positions of each test set review in these two calibration set lists determine the class assignment.

Below is a short illustrative demonstration example of how this recalibration is performed (the actual calibration lists sizes for both classes are much larger).

In Figure 3, the two separate lists of sorted calibration set class probabilities, one for each class, are shown to the left and right, and contain 19 and 9 predictions, respectively. In the upper-middle table, the predictions (class probabilities) for a set of five test reviews are shown. For each test review, the probability predictions for each class are then placed in the corresponding list of sorted calibration set class probabilities and the number of probabilities with lower or equal values are counted. For the first review, the value of 0.047 is therefore placed at the very end of the list while the value of 0.953 has seven lower values. Both lists have now been extended by 1 prediction, the test review to be determined, making them contain 20 and 10 predictions, respectively. The significance level (error rate) in this demonstration example is set to 0.2 (20% accepted errounous predictions). The ratios between the number of predictions with lower or equal probabilities to the new test review and the total number of predictions in each list are 0/20 ( =  0) and 7/10 ( =  0.7), respectively. Since the former ratio is lower than the significance level of 0.2, the prediction cannot be considered as neutral-positive. The latter ratio is, however, higher than 0.2 and the prediction can be considered as a negative class prediction. Thus, the review prediction will be classified as “negative.” The remaining four reviews are classified in same manner. Since the fourth review has ratios for both classes that are equal to the significance level, the prediction is then given for both classes (neutral-positive and negative) as an outcome (the both class). Notably, despite that the second review has a relatively high prediction probability for the neutral-positive class (0.892) and a relatively low prediction probability for the negative class (0.108), the review is still correctly classified as “negative” by the conformal prediction recalibration procedure despite the large-class imbalance that exists between the neutral-positive and negative class in this investigation.

Figure 3. Mondrian conformal prediction recalibration scheme.

For a binary classification task, the outcome for MCP can be one of the four possibilities: negative class, neutral-positive class, both (negative and neutral-positive) classes (the both class) or none of two classes (the empty class). For further description of this procedure, see Norinder et al., (2018). In MCP, a prediction is determined as correct if the prediction contains the correct class which means that both predictions are always correct since they, for binary classifications, always contain the correct class and empty predictions are always erroneous since no classes are included in the prediction. The latter is also an indication of that these predictions are outside the applicability domain of the model where reliable predictions can be made.

There are two key properties in MCP that need to be defined: Validity and efficiency. The validity (for each class in MCP), which is guaranteed by the mathematical proof (Vovk et al., 2005) given that the data are exchangeable, is defined as the percentage of correct predictions for each class. Efficiency is defined as the percentage of single-class, i.e. negative or neutral-positive, predictions irrespective of whether the prediction is correct or not. For the demonstration, example above the validity is 1.0 (100%) since all predictions are correct (the both prediction always being correct) while the efficiency is 0.8 (80%, four single-class predictions out of five).

3. Results (and Discussion)

The results from the analysis are presented in Table 2 and Figures 4 and 5. Inspection of Table 2 shows that all models are valid, given some small statistical fluctuations, or, for the most part, somewhat over-conservative, e.g. the error rate is smaller than the error rate (significance level) set by the user which is especially true for the majority, neutral-positive, class. The error rate where maximum efficiency, i.e. maximum single-class (neutral-positive or negative) predictions, occurs is around 12.5% for all datasets except for Books. The maximum efficiencies are very high with values between 97.4% and 99.9% both for the majority (Figure 4) and minority (Figure 5) classes, respectively. For Books all predictions are single-class (100% efficiency). Accuracy, in this case balanced accuracy (BA), calculated for single-class predictions, is high and is between 89.8 and 92.2% at the maximum efficiency error rate of 12.5%. Since efficiency is very high, the difference between calculated validity and BA is small and other, more traditional, measures useful for imbalanced datasets, e.g. Cohen’s Kappa and Matthews Correlation Coefficient, can also be inspected in order to assert the quality of the derived models. Also, these measures show high numbers indicative of models with significant predictive ability. Furthermore, measures such as sensitivity and specificity indicate significant predictive ability of the derived models with values between 0.90 and 0.97 and 0.88 and 0.96 for the former and latter measures, respectively. It should here also be noted that all of the reviews in the test set is of later, i.e. newer, date than the reviews in the training set and therefore the external test set predictions are of prospective nature which is many times considered more challenging than predicting from the same time span as that of the training set.

Figure 4. Test set efficiency for the neutral-positive majority class.

Figure 5. Test set efficiency for the negative minority class.

Table 2. External test set results from DNN/CP modelling.^a

category	Significance level	Validity neutral and positive class	Validity negative class	Efficiency total	Efficiency neutral and positive class	Efficiency negative class	Class empty	Class both	kappa	mcc	ba	se	sp
books	0.1	0.936	0.904	1	1	1	0	0	0.578	0.618	0.920	0.936	0.904
books	0.125	0.936	0.904	1	1	1	0	0	0.578	0.618	0.920	0.936	0.904
books	0.15	0.936	0.904	1	1	1	0	0	0.578	0.618	0.920	0.936	0.904
books	0.2	0.936	0.904	1	1	1	0	0	0.578	0.618	0.920	0.936	0.904
books	0.25	0.936	0.904	1	1	1	0	0	0.578	0.618	0.920	0.936	0.904
cds_and_vinyl	0.1	0.941	0.898	0.974	0.974	0.960	0	107,25	0.456	0.521	0.917	0.939	0.894
cds_and_vinyl	0.125	0.927	0.874	0.995	0.996	0.993	1854	0	0.423	0.492	0.906	0.932	0.88
cds_and_vinyl	0.15	0.914	0.854	0.970	0.971	0.954	12,114	0	0.466	0.529	0.918	0.942	0.894
cds_and_vinyl	0.2	0.887	0.813	0.926	0.927	0.888	30,188	0	0.547	0.596	0.936	0.957	0.915
cds_and_vinyl	0.25	0.858	0.766	0.884	0.886	0.821	47265	0	0.619	0.656	0.950	0.968	0.933
cell_phones_and_accessories	0.1	0.917	0.917	0.960	0.960	0.958	0	32,507	0.670	0.692	0.913	0.914	0.913
cell_phones_and_accessories	0.125	0.899	0.897	0.999	0.999	0.999	0	871	0.625	0.652	0.898	0.899	0.897
cell_phones_and_accessories	0.15	0.882	0.879	0.968	0.969	0.967	25,537	0	0.661	0.683	0.910	0.911	0.909
cell_phones_and_accessories	0.2	0.850	0.837	0.910	0.911	0.903	72,387	0	0.729	0.744	0.930	0.933	0.927
cell_phones_and_accessories	0.25	0.818	0.794	0.860	0.861	0.852	113,152	0	0.780	0.789	0.941	0.95	0.932
electronics	0.1	0.914	0.921	0.988	0.988	0.988	0	47,922	0.658	0.683	0.917	0.913	0.92
electronics	0.125	0.895	0.902	0.974	0.974	0.974	106,067	0	0.675	0.699	0.922	0.919	0.926
electronics	0.15	0.876	0.881	0.940	0.940	0.940	244,452	0	0.717	0.736	0.935	0.931	0.938
electronics	0.2	0.840	0.838	0.883	0.883	0.878	478,371	0	0.784	0.796	0.952	0.951	0.954
electronics	0.25	0.804	0.794	0.835	0.836	0.824	675,195	0	0.828	0.836	0.963	0.962	0.964
movies_and_tv	0.1	0.934	0.910	0.970	0.971	0.965	0	49,424	0.677	0.697	0.919	0.932	0.906
movies_and_tv	0.125	0.918	0.890	0.996	0.996	0.996	5978	0	0.641	0.665	0.908	0.922	0.894
movies_and_tv	0.15	0.904	0.870	0.967	0.968	0.960	54,758	0	0.684	0.703	0.920	0.934	0.907
movies_and_tv	0.2	0.877	0.829	0.918	0.921	0.898	134,762	0	0.752	0.764	0.938	0.953	0.923
movies_and_tv	0.25	0.852	0.787	0.879	0.883	0.844	199,736	0	0.803	0.810	0.949	0.965	0.933
office_products	0.1	0.932	0.910	0.952	0.953	0.934	0	25,994	0.574	0.614	0.916	0.929	0.904
office_products	0.125	0.917	0.888	0.989	0.989	0.983	0	6180	0.529	0.575	0.901	0.916	0.886
office_products	0.15	0.902	0.865	0.981	0.981	0.973	10,393	0	0.540	0.584	0.904	0.92	0.889
office_products	0.2	0.874	0.823	0.931	0.933	0.903	37,614	0	0.605	0.640	0.924	0.937	0.912
office_products	0.25	0.847	0.783	0.889	0.892	0.846	59,968	0	0.657	0.685	0.937	0.949	0.926

^a kappa = Cohen’s kappa; mcc = Matthews Correlation Coefficient; ba = balanced accuracy; class empty = number of predicted reviews in the empty class; class both = number of predicted reviews in the both class; se = sensitivity; sp = specificity.

However, the described procedure of data preparation and machine learning does not, in its present form, seem to be able to handle all kinds of imbalanced review types. For datasets with a low median number of words per review, somewhere below 70 words for negative reviews, the proposed procedure is not able to establish a model but simply returns all predictions as if they were neutral-positive, i.e. the majority class (Table 1; successful = no). Several modifications to the protocol have been investigated, such as enlarging the vocabulary size, different sizes of nodes in the DNN layers, various over- and/or under-sampling techniques in order to balance the two classes and weighting schemes both with respect to class, but also reviews, without success. It should be noted that for the cell phones and accessories dataset, a successful model was developed although the set has a low median number of words per review, below 70 words for the negative reviews, and vice versa for the musical instruments dataset where no model was successfully created. A likely explanation for these results is that the former dataset has the highest percentage of negative reviews (12.9%) of all the studied categories, while the opposite is true for the latter dataset where only 6.2% of reviews are negative making it the second lowest investigated category.

Deep learning has been successfully used for sentiment analysis of Amazon data, as previously referred to, using word vectors and a recurrent neural network of almost 5.2 million reviews using a training set based on four categories (beauty, book, electronic and home) (Shrestha & Nasoz, 2019), indicating some level of generalizability between categories. Therefore, the unsuccessful datasets (Table 1) were predicted by two models from successful datasets (Movies_and_TV and Cds_and_Vinyl, respectively) using both the models calibration set but also using the calibration set of the investigated (unsuccessful) dataset in order to derive the conformal prediction p-values and, in turn, the outcome (classification) from the prediction. Figures 1 and 2 depict how the calibration set is generated and used for model building and prediction. The average results of these predictions are shown in Table 3.

Table 3. Mean values for validity and efficiency across unsuccessful datasets using cross-category models.

Significance level	model&calibr set^a	Mean validity neutral and positive class	Mean validity negative class	Mean efficiency neutral and positive class	Mean efficiency negative class	Mean sensitivity	Mean specificity
0.1	cds_cds_cal	0.913	0.840	0.962	0.945	0.910	0.830
0.125	cds_cds_cal	0.894	0.809	0.996	0.993	0.898	0.814
0.15	cds_cds_cal	0.875	0.779	0.958	0.938	0.913	0.830
0.2	cds_cds_cal	0.837	0.723	0.897	0.845	0.932	0.855
0.25	cds_cds_cal	0.796	0.683	0.844	0.778	0.943	0.877
0.1	cds_own_cal	0.927	0.906	0.868	0.850	0.915	0.889
0.125	cds_own_cal	0.909	0.887	0.915	0.905	0.900	0.875
0.15	cds_own_cal	0.891	0.866	0.959	0.953	0.886	0.859
0.2	cds_own_cal	0.855	0.827	0.970	0.967	0.881	0.856
0.25	cds_own_cal	0.818	0.786	0.906	0.896	0.903	0.877
0.1	movies_movies_cal	0.916	0.875	0.958	0.946	0.912	0.867
0.125	movies_movies_cal	0.895	0.845	0.995	0.994	0.900	0.850
0.15	movies_movies_cal	0.873	0.816	0.953	0.940	0.915	0.868
0.2	movies_movies_cal	0.828	0.760	0.883	0.852	0.937	0.892
0.25	movies_movies_cal	0.775	0.702	0.813	0.772	0.953	0.910
0.1	movies_own_cal	0.925	0.909	0.900	0.897	0.916	0.899
0.125	movies_own_cal	0.906	0.887	0.954	0.949	0.902	0.881
0.15	movies_own_cal	0.887	0.866	0.988	0.987	0.890	0.868
0.2	movies_own_cal	0.848	0.823	0.933	0.929	0.909	0.887
0.25	movies_own_cal	0.805	0.783	0.870	0.866	0.926	0.905

^a Models and calibration set: cds_cds_cal = cds and vinyl’s model using cds and vinyl’s calibration set; cds_own_cal = cds and vinyl’s model using the predicted dataset’s own calibration set; movies_movies_cal = movies and TV’s model using movies and TV’s calibration set; movies_own_cal = movies and TV’s model using the predicted dataset’s own calibration set.

The cross-category predictions for all of the previously unsuccessful datasets now result in highly predictive estimates both with respect to validity as well as efficiency at various significance levels. Not surprisingly, using the calibration set from the same review category as the predicted test dataset for predicting the outcome (class) is better than using the original model’s calibration set that is related to a different review category. This is especially noticeable for the more sensitive negative minority class where using the original model’s calibration set does not result in valid models, i.e. the observed error rate is larger than the error rate (significance level) set by the user (Table 3). The average efficiency is slightly lower, but still at a very high level, using cross-category predictions when compared to the reported “successful” in-category predictions (Table 2), in which the derived model and the predicted dataset belong to the same category. Also, sensitivity and specificity for cross-category predictions are high with values of 0.89 and 0.86, respectively. The significance level for maximum efficiency is shifted from around 0.125 for the previously reported in-category (successful) predictions to 0.15 for the cross-category ones which is not surprising since 100% generalizability from one category to another cannot be expected. However, since the shift is very small, only 0.025 (an increase of the error rate by 2.5%), the cross-category generalizability seems to be rather significant for this type of binary classification indicating only a very small portion of information loss.

How can the outcome from these models be interpreted in terms of important sets of words for neutral-positive and negative reviews, respectively?

Floh et al. (2013) established that online reviews consist of two dimensions, valence which describes the category of review on the positive to negative range and intensity which determines the level between high and moderate valence. This definition of “valence” is similar to the term sentiment, both referring to the consumers’ opinion, and hence the terms “sentiments” and “sentiment intensity” are used in the present study for descriptive purposes. “Sentiment intensity” in online reviews can differ regarding the positive or negative intensity in the words that are used in the review, but may also differ in their overall expression. Reviews in the negative range seem to focus on the perception and feelings connected to the reviewers’ internal disposition, while reviews on the positive range are more related to the product (externally connected) (Kim & Gupta, 2012).

The results of the present investigation indicate that the sentiment intensity of the neutral-positive reviews seem more consistent in which words the reviewers use. Words such as “awesome,” “fantastic,” and “superb” place high in terms of importance in the models and are occurring regardless of product category. As the neutral-positive expression seems to focus on the product, it is possible that these types of expressions are more uniform regardless of whether the review concerns the product category of e.g. Books or Electronics. Words in the negative reviews seem to spread over a larger set of words, where the “sentiment intensity” in such words is difficult to identify as negative reviews are more diverse in their expressions in comparison to the positive reviews. One of the major motivations for leaving an online review is the expression of emotions (Ismagilova et al., 2020). However, eWOM, as opposed to traditional word-of-mouth, lacks the ability to communicate non-verbally with cues such as gestures and facial expressions (Kim & Gupta, 2012). Since eWOM lacks these cues, reviewers therefore become more reliant on using more explanatory expressions. Negative reviews focusing on the internal disposition of the reviewer, where a larger set of words formed together in a longer expression is needed in order to express a negative sentiment, could explain why a consistency in a set of more specific words, like the frequently occurring terms in neutral-positive reviews, are more difficult to identify using the currently employed deep learning LSTM architecture.

The results presented in Tables 2 and 3 show that the present approach using deep learning in combination with conformal prediction is capable of handling the existing class imbalances in the investigated datasets.

From a methodological point of view, there are several new contributions in this work. The conformal prediction framework presented in this investigation is based on a mathematical proof (Vovk et al., 2005). This means that the acceptable error rate (percentage errors) from model predictions can be set by the user in relation to what is needed for the decision to be taken from the outcome of these predictions. The requirement is that the investigated data are exchangeable. This requirement does not, however, constitute an additional criterion for conformal prediction but is needed for all empirical modelling namely that the new data to be predicted behave like the old data on which the model was created, i.e. both new and old data have the same underlying relationship. The Mondrian version of conformal prediction used in this investigation has the important advantage in that the error rate can be set (controlled) for both classes individually. This is especially important in order to ensure adequate handling and predictive performance for the minority class in highly imbalanced datasets. Furthermore, the graceful handling of the minority class is performed without any class balancing, e.g. under- and/or over-sampling or weighing, schemes since such measures may create issues when modelling imbalanced datasets. Also, the outcomes of the predictions provide additional information to the user as well as of the limitations of the current model. If for instance, at a certain significance level (error rate), the model returns a large percentage of both predictions, i.e. the efficiency is low, this indicates that the model, at present, lacks sufficient information. This, in turn, may then result in an upgrade of the model where additional, new, information has been added. If the model instead returns a large percentage of empty predictions, i.e. that no class could be assigned, this indicates that many of the new predictions (reviews) are very different from the reviews on which the present model was built and outside the applicability domain of the model. In this case, the model should be upgraded, once the outcome (class) has been obtained for some of these empty predictions, in order to enlarge the applicability domain of the model for future predictions. Furthermore, the determination of the applicability domain of the model is an intrinsic part of conformal prediction and handled within the framework without any additional measures.

4. Conclusions

In this investigation, we have shown that the combination of deep learning and conformal prediction results in highly predictive and efficient temporal test set sentiment estimates for categories of product reviews with sufficient average word count (∼100 words or more) where the error rate from the derived models can be controlled. For categories of product reviews with insufficient average word count, the present work indicates that cross-category predictions, i.e. using a model of another category for predictions in combination with the calibration set from the investigated review category, also seem to be highly predictive and efficient with only minor loss in terms of efficiency indicating generalizability across product categories. The investigation also shows that the combination of deep learning and conformal prediction gracefully handles class imbalances without explicit class balancing measures. Thus, the mathematical rigor and performance of Mondrian conformal prediction shown in this work makes the framework, agnostic to the machine learning technique employed, an attractive alternative to more traditional approaches for sentiment analysis and other similar tasks.

Disclosure statement

No potential conflict of interest was reported by the author(s).