Analyzing public sentiments online: combining human- and computer-based content analysis

ABSTRACT

Recent technological developments have created novel opportunities for analyzing and identifying patterns in large volumes of digital content. However, many content analysis tools require researchers to choose between the validity of human-based coding and the ability to analyze large volumes of content through computer-based techniques. This study argues for the use of supervised content analysis tools that capitalize on the strengths of human- and computer-based coding for assessing opinion expression. We begin by outlining the key methodological issues surrounding content analysis as performed by human coders and existing computational algorithms. After reviewing the most popular analytic approaches, we introduce an alternative, hybrid method that is aimed at improving reliability, validity, and efficiency when analyzing social media content. To demonstrate the usefulness of this method, we track nuclear energy- and nanotechnology-related opinion expression on Twitter surrounding the Fukushima Daiichi accident to examine the extent to which the volume and tone of tweets shift in directions consistent with the expected external influence of the event. Our analysis revealed substantial shifts in both the volume and tone of nuclear power-related tweets that were consistent with our expectations following the disaster event. Conversely, there was decidedly more stability in the volume and tone of tweets for our comparison issue. These analyses provide an empirical demonstration of how the presented hybrid method can analyze defined communication sentiment and topics from large-scale social media data sets. The implications for communication scholars are discussed.

KEYWORDS:

• Content analysis
• sentiment analysis
• human-based coding
• computer-based coding
• supervised machine learning
• Twitter

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Leona Yi-Fan Su is the corresponding author. She is a PhD candidate in the Department of Life Sciences Communication at the University of Wisconsin-Madison. Her research interests focus on the interplay between new media and society, particularly in the context of science and the environment, and on how the new media influence public opinion and understanding [email: [email protected]].

Michael A. Cacciatore (PhD, University of Wisconsin-Madison, 2013) is an Assistant Professor in the Department of Advertising and Public Relations at the University of Georgia. His research focuses on risk communication with an emphasis on media coverage of risk and opinion formation for risk topics [email: [email protected]].

Xuan Liang has her PhD (2015) in mass communications from the University of Wisconsin-Madison. Her research examines the intersection of science, media, and society. She is particularly interested in media representations of scientific and environmental issues, scientists’ public communication activities, and the role of new media and individual’s values in the formation of public opinion about scientific and environmental issues [email: [email protected]].

Dominique Brossard is Professor and Department Chair in the Department of Life Sciences Communication at the University of Wisconsin-Madison and an affiliate of the UW-Madison Robert & Jean Holtz Center for Science and Technology Studies, the UW-Madison Center for Global Studies, and the Morgridge Institute for Research. Her research agenda focuses on the intersection between science, media, and policy, with a focus on online environments [email: [email protected]].

Dietram A. Scheufele is the John E. Ross Chair in Science Communication and Vilas Distinguished Achievement Professor at the University of Wisconsin-Madison and in the Morgridge Institute for Research. His research focuses on public opinion, political communication, and public attitudes toward new technologies [email: [email protected]].

Michael A. Xenos is Communication Arts Partners Professor and Department Chair in the Department of Communication Arts at the University of Wisconsin-Madison. His research focuses on how individuals, political candidates, journalists, and other political actors adapt to changes in information and communication technologies, and how these adaptations affect broader dynamics of political communication and public deliberation [[email protected]].

Notes

1. The details behind these steps can be found in our section entitled ‘Methodology.’

2. In comparison with Twitter's sampled application program interface (API) service that allows retrieval of no more than 10% of all the public tweets (boyd & Crawford, 2012), the Twitter Firehose feed typically grants full access to 100% of public tweets (Morstatter, Pfeffer, Liu, & Carley, 2013).

3. Our analysis revealed a sharp increase in the volume of nuclear energy-related tweets beginning at the onset of the Fukushima Daiichi disaster. Perhaps of even greater importance, our sentiment analysis revealed that the gap between pessimistic and optimistic opinions about nuclear energy grew sharply at that time, and remained wide for several months following the event. In concert with the negative coverage of nuclear power on traditional news during the period (Friedman, 2011), the increase in pessimistic opinions was decidedly certain, and was contextualized through the lens of environment, health, and safety-related concerns.

Additional information

Funding

This work was supported by grants from the Nanoscale Informal Science Education Network (Grant No. DRL-0940143) and the UW-Madison Nanoscale Science and Engineering Center in Templated Synthesis and Assembly at the Nanoscale (Grant No. DMR-0832760). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.