Rating the Valence of Media Content about Electronic Cigarettes Using Crowdsourcing: Testing Rater Instructions and Estimating the Optimal Number of Raters

ABSTRACT

Electronic cigarettes (e-cigarettes) are a controversial public health topic due to their increasing popularity among youth and the uncertainty about their risks and benefits. Researchers have started to assess the valence of media content about e-cigarette use, mostly using expert coding. The current study aims to offer a methodological framework and guideline when using crowdsourcing to rate the valence of e-cigarette media content. Specifically, we present (1) an experiment to determine rating instructions that would result in reliable valence ratings and (2) an analysis to identify the optimal number of raters needed to replicate these ratings. Specifically, we compared ratings produced by crowdsourced raters instructed to rate from several different perspectives (e.g., objective vs. subjective) and determined the instructions that led to reliable ratings. We then used bootstrapping methods and a set of criteria to identify the minimum number of raters needed to replicate these ratings. Results suggested that when rating e-cigarette valence, instructing raters to rate from their own subjective perspective produced reliable results, and nine raters were deemed the optimal number of raters. We expect these findings to inform future content analyses of e-cigarette valence. The study procedures can be applied to crowdsourced content analyses of other health-related media content to determine appropriate rating instructions and the number of raters.

Notes

1. The qualification task included six texts (e-cig irrelevant [n = 1], pro-ecig [n = 2], anti-ecig [n = 2], or not applicable [n = 1]). We qualified raters based on whether they correctly answered valence or relevance for the five texts that were clear in valence/relevance (i.e., e-cig irrelevant, pro-ecig and anti-ecig texts).

2. The proportion of qualified participants was 49% on average and did not differ by condition (χ² (3) = 2.93, p = .40).

3. To identify e-cigarette-related texts, we compiled a list of e-cigarette keyword rules, which were developed through expert consensus and an iterative process of adding and dropping keywords with a training set of broadly tobacco-related texts based on precision and recall. Final precision for e-cigarette texts was >.97 for all sources and recall was 1.0. The final set of keywords including e(-)cig, e(-)cigarette, electronic cigarette, e(-)hookah, hookah pen, vape, vaping, blu, njoy, ego, etc., were used to retrieve e-cigarette-related articles and determine whether an article was mostly about e-cigarettes. The three mentions cutoff demonstrated the precision of .85 and recall of .80 in distinguishing between more than passing and passing mention texts across different types of media.

4. We acknowledge that raters can rate smaller units of text such as sentences or paragraphs. The present study, however, focuses on the entire text as the unit of analysis because (1) it allows raters to rate in a more naturalistic setting (i.e., readers are expected to understand entire texts as a whole vs. interpreting disconnected chunks of text individually); and (2) because it allows assessment of the general impression of the text which dovetails with the overall project goal to predict the public’s intentions from analyzed content.

5. Determining the within-condition consensus for each text followed the rule used to determine the “majority valence” label of texts described next in the analysis of the number of raters needed.

6. When the consensus for a certain text within a certain condition was NA, for those who rated the text as either pro-ecig or anti-ecig, the proportion of raters choosing either pro-ecig or anti-ecig was used to judge the consistency score for these raters. For example, if the proportion of those choosing pro-ecig was larger than anti-ecig, those who rated the text as pro-ecig received a score of 2 while those who rated it as anti-ecig were assigned a score of 1.

Additional information

Funding

The research reported in this publication was supported by the National Cancer Institute (NCI) of the National Institutes of Health (NIH) and FDA Center for Tobacco Products (CTP) under award [P50CA179546] and award [R25CA057711] from the NCI of the NIH. The content is solely the responsibility of the author and does not necessarily represent the official views of the NIH or the Food and Drug Administration (FDA).