Reassessing the Facebook experiment: critical thinking about the validity of Big Data research

ABSTRACT

The Facebook experiment of 2014 manipulated the contents of nearly 700,000 users’ News Feeds to induce changes in their emotions. This experiment was widely criticized on ethical grounds regarding informed consent. This controversy, however, diverted attention from a more important concern the experiment was intended to address, which is the impact of Facebook use on well-being. In this paper, I explore the well-being concerns raised by prior research and argue that the experiment does not alleviate them, owing to poor research design. As the question of Facebook's impact on well-being is of great importance, both to Facebook and to society overall, there is a pressing need for more experimental research that is both sensitive to informed consent and carefully designed to yield reliable results. In turn, the lessons of this case have implications for general issues of validity that emerge in Big Data research, now in vogue at major scientific venues.

KEYWORDS:

• Facebook experiment
• social media
• well-being
• Big Data
• LIWC

Acknowledgments

I thank my advisor, Steve Weber, colleagues in my department's Doctoral Research and Theory Workshop, consultants in Berkeley's D-Lab and my reviewers for their helpful feedback on this paper.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes on contributor

Galen Panger is a doctoral candidate in the School of Information at the University of California, Berkeley, and is co-director of the Berkeley Center for Technology, Society & Policy. His research interests include the use of social media data to understand well-being and public opinion, social media psychology and digital citizenship. [email: [email protected]]

Notes

1. The controversy centred on the study's lack of informed consent procedures. For reference, see Grimmelmann (2014) or refer to the archived front page of Techmeme, a technology news aggregator, from June 29, 2014 (http://www.techmeme.com/140629/h1410).

2. See ‘About PNAS', http://www.pnas.org/site/aboutpnas/index.xhtml

3. Burke finds that broadcasting is associated with lower life satisfaction (Burke, 2011) and marginally higher stress (Burke & Kraut, 2013).

4. To estimate effects, Burke uses a linear multilevel model with a lagged dependent variable. Though observational, this model produces estimates that are less biased than cross-sectional models, ‘in effect controlling for an individual's previous level of the outcome variable (e.g. stress) and all of the unmeasurable factors that contribute to it’ (Burke & Kraut, 2013, p. 1424).

5. KGH provide no further information about the average or distribution of the percentage of posts removed from News Feed. They maintained two control groups because positive and negative posts occur at different rates in News Feed and thus removing a given percentage of posts involves removing a different number in the positive experimental and control groups than the negative experimental and control groups.

6. In this scenario, negative posts increase in proportion when any percentage (greater than zero) of positive posts is removed. This is demonstrated mathematically by solving the following equation for x, which represents the percentage of positive posts removed:The values 0.468, 0.224 and 0.308 stand for the exact percentages of positive, negative and neutral posts, respectively, rather than the rough percentages used in the example above. A similar exercise demonstrates that positive posts increase in proportion when any percentage greater than zero of negative posts is removed. Note that this scenario does not account for the possibility that some posts may be classified as containing both positive and negative emotion, and thus may be removed in either experimental condition. This possibility is not discussed in the Facebook experiment but was acknowledged in an email from Kramer, who did not know the percentage of posts that received this dual classification (personal communication, 23 June 2015). Because posts may be both positive and negative, removing one emotion from News Feed may actually decrease the proportion of the other, rather than increase it, if a large enough percentage of posts receive dual classification. However, because removing one emotion from News Feed resulted in subjects producing more of the other, I proceed under the assumption that the other likely also increased in proportion in News Feed. It is possible that, by chance, the percentage of posts receiving dual classification is exactly the percentage needed to hold one emotion constant in News Feed when the other is reduced, but this does not seem likely.

7. Though KGH argue that mimicry cannot account for the ‘cross-emotional encouragement effect', they do not explain what can or why the effect is happening absent an internal validity problem. Elsewhere, the authors argue for the independence of positive and negative emotion, stating that ‘positivity and negativity were evaluated separately given evidence that they are not simply opposite ends of the same spectrum. Indeed, negative and positive word use scarcely correlated’ (2014, p. 8789). If positive and negative emotion operate independently, then there is no a priori reason to believe less of one in News Feed should cause more of the other in subjects’ posts. Again, a plausible explanation for this is the internal validity problem. However, even if positive and negative emotion are dependent, and less of one generally implies more of the other, then there is also no reason, and none is offered, why mimicry and conformity would not also show this dependent relationship.

8. Facebook posts do now have a simple optional mood indicator, however (see Constine, 2013).

9. Kramer (2010) correlates scores on the Satisfaction With Life scale with LIWC analysis of an average of 244 status updates per subject (n = 1341 subjects).

10. Such a resharing button was likely present when the study was conducted (e.g. see video in Tow, 2011).

11. For more on these trade-offs and how they are typically addressed, see Hektner et al. (2007, especially pages 50–58).

12. Mobile push notifications would actually enable researchers to solicit mood reports throughout the day, on or off Facebook, providing a more complete view over time of the emotional impact of experimental interventions.