The Effect of Rating Scale Design on Extreme Response Tendency in Consumer Product Ratings

ABSTRACT

Consumer-generated product ratings have become a fundamental component of online businesses, yet seem to be susceptible to biases that shift the distribution toward the extreme points of the rating scales. Response biases (such as extreme response style) due to intrinsic traits are widely investigated in survey design and marketing research, yet little is known about how rating scale variations in consumer-generated product evaluation systems influence the generated ratings. In two experiments across two product categories, the findings suggest that explicitly labeling the midpoint of the rating scale reduces extreme responses, though this effect is attenuated when emotional labels are used at the endpoints. Furthermore, the use of emotional labels deters users from giving extreme ratings when the size of the rating scale is large. The effects remain consistent across different model specifications and robustness checks. The study extends the theoretical implications regarding extreme response bias in the context of consumer product ratings and enriches the understanding of response behavior beyond self-selection reasons, and across various rating scale elements, product contexts, and sample traits. The broad use of product ratings in generating recommendations and estimating future product performance and investments necessitates a focus on how to better account for such potential distortions in these ratings due to the implemented rating scales.

KEY WORDS AND PHRASES:

• Product rating scales
• response bias
• response style
• word of mouth

Acknowledgments

The author would like to thank Erasmus Research Institute of Management (ERIM) for financial support for part of this research. The author also thanks Sanne van Rennes for assisting with the data collection as well as participants at the 2015 European Conference of Information Systems for valuable feedback on an earlier version of this paper.

Notes

^1. Facebook changed the labels of the 5-point scale from “really don’t like it” / “love it” to “Very Poor” / “Excellent” (adweek.com/socialtimes/facebook-tests-new-rating-scales-for-places/293045); Youtube changed the five-point star rating system (“Poor” to “Awesome”) to a binary scale (“Like” to “Dislike”) (youtube-global.blogspot.com/2010/01/video-page-gets-makeover.html.

^2. Ratings were extracted on March 15, 2016.

^3. See Alexa.com. Furthermore, purchasing bias [32] is demoted because no verified purchase is required in order to rate the movies across these websites.

^4. For example, a five-point scale offers two gradation points (1–2 or 4–5) from the midpoint (3) whereas a nine-point scale offers four gradation points (1–4 or 6–9) from the midpoint (5).

^5. The movies were: Last Vegas, Ender’s Game, and Free Birds. The study was conducted in October 2013 and the movies were about to be released at the earliest in December 2013 or January 2014.

^6. The ratings were extracted in November 2014. The movies Last Vegas, Ender’s Game, and Free Birds had an average of 6.7, 6.8, and 6.2, respectively on IMDB.com (all rating options had at least 500 votes).

^7. In this study, emotional labels in a rating scale refer, for example, from “hate” to “love” a product (see Amazon and Netflix) whereas nonemotional labels (informational) refer, for example, to “Very poor” to “Excellent” products (see iTunes and Facebook).

^8. The following points were assigned to their answers: 0 = not aware; 1= aware but had not seen; 2 = had seen the movie. Knowledge was measured as the sum of the points from the list of the fifteen movies. The movies used in this task were all blockbusters, yet varied in terms of release year and movie genre.

^9. A seven-point scale was implemented to maintain a symmetric distance to both large and small scale sizes in the main task: a two-point difference from a five-point or nine-point scale). The items used were: (a) I work very hard most of the time, (b) I eat more than I should, (c) a college education is very important for success in today’s world, (d) I like to visit places that are totally different from my home, (e) investing in the stock market is too risky.

^10. The two chairs received an average of 3.2 and 4.8, respectively, out of 5.0 on Amazon.com (out of fifty reviews). This difference was reflected in the average rating in the current study (0.36 vs. 0.46 out of 1.0, respectively)

^11. A 0–10 scale was used because (a) it is more frequently observed in rating websites (compared to the 1–9 scale of Study 1) and (b) it is an odd-numbered scale with a midpoint.

^12. Participants who failed the respective attention checks (N = 52), used a mobile device (due to visual restrictions of the study) (N = 27), and had never visited any review website (N = 2) were excluded from the analysis.

Additional information

Notes on contributors

Dimitrios Tsekouras

DIMITRIOS TSEKOURAS ([email protected]) is an assistant professor of business information management at Rotterdam School of Management, Erasmus University, Netherlands. He received his Ph.D. in marketing from Erasmus School of Economics. His research interests revolve around consumer decision making and the role of information technology in decision assistance, product recommendations, and electronic word of mouth. His work has been included for presentation at several international conferences: INFORMS Marketing Science, International Conference on Electronic Commerce, International Conference of Information Systems, Conference on Information Systems and Technology, European Conference on Information Systems, Academy of Management, and others.