Ethical, political and epistemic implications of machine learning (mis)information classification: insights from an interdisciplinary collaboration between social and data scientists

Figures & data

Figure 1. Steps in the construction of a ML misinformation detection model. (1) Problem definition: design of strategy based on hypotheses, definitions and theories about how to identify misinformation. (2) Selection of multimodal inputs and outputs to be included into the classification model. (3) Ground-truthing: the ground truth dataset is used to train the model. A subset of the this is reserved for model validation. (4) Model validation: a subset of the ground truth dataset is used to test the model’s performance. Metrics of performance accompany the publication of classification models. (5) Model deployment: model outputs inform online content moderation decisions such as banning, downranking or flagging. The dotted arrows represent feedback loops between steps.

Table 1. A summary of cautions and contingencies in the development of ML misinformation classification models and their implications.

Steps	Cautions and contingencies	Implications
Problem definition	Researchers use different theories and definitions of misinformation. Problematisation relies on previous empirical studies and experiments. Researchers often use hypotheses of how misinformation manifests online.	Misinformation and truth are conceptualised as classifiable based on salient features and prior knowledge. Definitions and provisional hypotheses translate into intractable variables and norms embedded in a model.
Choice of inputs and outputs	The identification of inputs (e.g. text, image, a combination of both) and outputs (‘true’, ‘false’, ‘misleading’) is informed by a combination of technical and cost factors for accessing data.	Opportunity and feasibility constrain the choices of variables and how the problem is abstracted into a mathematical formalism. Proxy variables may be used due to reduced cost (e.g. humour as a signal of misinformation).
Ground-truthing	Ground truth datasets are temporally contingent and have diminishing returns vis à vis new events. Researchers might standardise labels to reduce complexity. Data annotation and labelling are susceptible to errors, partiality and uncertainty (e.g. factcheckers disagreeing on the verdict of a claim). Using fact-checks to curate datasets could overrepresent debunked claims over truthful claims.	Difficulty to check for claims relating to new events. Inconsistency among data sources leading to loss of nuance or amplification of errors. Models are better at recognising debunked claims and less so at checking truths. Selection biases influence what types of information are worthy of checking and which narratives are prioritised over others.
Model Validation	Performance evaluations are only indicative of novelty and progress among competing models. Benchmarks of performance against other models could be used across content domains.	Performance metrics influence moderation decisions such as censoring, downranking, flagging pieces of information. Model high performance can be a misleading metric of a model’s utility in particular topics and cultural contexts.
Model deployment	Moderation decisions can be informed on metrics of ML model performance. A logic of scaling up could lead to blanket decisions across cultures, locations, idiosyncrasies, laws. Utilitarian approach: some error or harm (false positives and false negatives) is acceptable.	High performing models leading to less human input. Using high performance models as a solution might underplay the responsibilities of platforms. Models could have uneven impacts across different contexts. Burden of algorithmic failures and harm shifted away from platforms e.g. appealing processes