A Model of Text for Experimentation in the Social Sciences

ABSTRACT

Statistical models of text have become increasingly popular in statistics and computer science as a method of exploring large document collections. Social scientists often want to move beyond exploration, to measurement and experimentation, and make inference about social and political processes that drive discourse and content. In this article, we develop a model of text data that supports this type of substantive research. Our approach is to posit a hierarchical mixed membership model for analyzing topical content of documents, in which mixing weights are parameterized by observed covariates. In this model, topical prevalence and topical content are specified as a simple generalized linear model on an arbitrary number of document-level covariates, such as news source and time of release, enabling researchers to introduce elements of the experimental design that informed document collection into the model, within a generally applicable framework. We demonstrate the proposed methodology by analyzing a collection of news reports about China, where we allow the prevalence of topics to evolve over time and vary across newswire services. Our methods quantify the effect of news wire source on both the frequency and nature of topic coverage. Supplementary materials for this article are available online.

KEYWORDS:

• Causal inference
• Experimentation
• High-dimensional inference
• Social sciences
• Text analysis
• Variational approximation

Supplementary Materials

The supplementary materials contain the article's appendix. Additionally, replication materials are available on Dataverse and can be found at http://dx.doi.org/10.7910/DVN/SIGIAU.

Acknowledgments

The authors thank Ryan Adams, Ken Benoit, David Blei, Patrick Brandt, Amy Catalinac, Sean Gerrish, Adam Glynn, Justin Grimmer, Gary King, Christine Kuang, Chris Lucas, Brendan O’Connor, Arthur Spirling, Alex Storer, Hanna Wallach, Daniel Young, and in particular Dustin Tingley, for useful discussions, and the editor and two anonymous reviewers for their valuable input. The first two authors contributed equally to this work.

Additional information

Funding

This research supported, in part, by The Eunice Kennedy Shriver National Institute of Child Health & Human Development under grant P2CHD047879 to the Office of Population Research at Princeton University, by the National Science Foundation under grants CAREER IIS-1149662, and IIS-1409177, and by the Office of Naval Research under grant YIP N00014-14-1-0485 to Harvard University. This research was largely performed when Brandon M. Stewart was a National Science Foundation Graduate Research Fellow at Harvard University. Edoardo M. Airoldi is an Alfred P. Sloan Research Fellow. The content is solely the responsibility of the authors and does not necessarily represent the official views of theNIH, of the NSF, nor of the ONR.