Categorizing political campaign messages on social media using supervised machine learning

ABSTRACT

Scholars have access to a rich source of political discourse via social media. Although computational approaches to understand this communication are being used, they tend to be unsupervised and off-the-shelf algorithms to describe a corpus of messages. This article details our approach at using human-supervised machine learning to study political campaign messages. Although some declare this technique too labor-intensive, it provides theoretically informed classification, making it more accurate and reliable. This article describes the design decisions and accuracy of our algorithms, and the applicability of the approach to classifying messages from Facebook and Twitter across two cultures and to advertisements.

KEYWORDS:

• Supervised machine learning
• political campaigns
• content analysis
• computational social science

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Data are available from the authors upon request. All scripts for training and evaluating a BERT model are available at: https://github.com/Syracuse-CCDS/Illuminating_Training_Testing.

Additional information

Funding

This work was supported by a John S. and James L. Knight grant, and a Fellowship from the Tow Center for Digital Journalism at Columbia University.

Notes on contributors

Jennifer Stromer-Galley

Jennifer Stromer-Galley (Ph.D., 2002, University of Pennsylvania) is Professor in the School of Information Studies, Senior Associate Dean for Academic and Faculty Affairs, and Director of Diversity, Equity, Inclusion, and Accessibility Initiatives. She is a former president of the Association of Internet Researchers. Her book Presidential Campaigning on the Internet Age received the 2015 Roderick P. Hart Top Book Award in the Political Communication Division of the National Communication Association. Jenny has been studying “social media” since before it was called social media, studying online interaction and strategic communication in a variety of contexts, including political forums and online games. She has published over 70 journal articles, proceedings, and book chapters, and received over $15 million in federal and corporate grants to support her research endeavors.

Patricia Rossini

Patrícia Rossini (Ph.D., 2017, Federal University of Minas Gerais) is a Senior Lecturer in Communication, Media & Democracy at the University of Glasgow. Prior to joining UofG, she was an inaugural Derby Fellow in the Department of Communication and Media at the University of Liverpool (2019-22), and a post-doctoral researcher at the School of Information Studies (iSchool) at Syracuse University (2017-19). Patrícia studies the interplay between political communication and technologies, with a focus on digital threats to democracy—specifically, uncivil and intolerant online discourse, mis- and disinformation, as well as (dark) participation, democratic backsliding, and online campaigns. Her research has been funded by social media companies such as Facebook, Google, Twitter, and WhatsApp; the British Academy, and the Knight Foundation (USA).