Screenomics: A Framework to Capture and Analyze Personal Life Experiences and the Ways that Technology Shapes Them

Abstract

Digital experiences capture an increasingly large part of life, making them a preferred, if not required, method to describe and theorize about human behavior. Digital media also shape behavior by enabling people to switch between different content easily, and create unique threads of experiences that pass quickly through numerous information categories. Current methods of recording digital experiences provide only partial reconstructions of digital lives that weave – often within seconds – among multiple applications, locations, functions, and media. We describe an end-to-end system for capturing and analyzing the “screenome” of life in media, i.e., the record of individual experiences represented as a sequence of screens that people view and interact with over time. The system includes software that collects screenshots, extracts text and images, and allows searching of a screenshot database. We discuss how the system can be used to elaborate current theories about psychological processing of technology, and suggest new theoretical questions that are enabled by multiple timescale analyses. Capabilities of the system are highlighted with eight research examples that analyze screens from adults who have generated data within the system. We end with a discussion of future uses, limitations, theory, and privacy.

Notes

¹ N = 30 undergraduate students (22 women, 8 men) age 19 to 23 years from a medium-sized university in the western US; 4 days of screenshots.

² N = 1, 24 h of screenshots provided by a Hispanic, young adult female living in the United States.

³ N = 1, 36 h of screenshots provided by a White, young adult male living in the United States.

⁴ N = 30 undergraduate students (22 women, 8 men) age 19 to 23 years from a medium-sized university in the western US; 4 days of screenshots.

⁵ Story 1 is based on analysis of 24 h of screenshots provided by a White, middle-aged adult male living in the United States; Story 2 is based on analysis of 24 h of screenshots provided by a young adult male living in Myanmar.

⁶ N = 1, 24 h of screenshots provided by a White, young adult female college student living in the United States.

⁷ N = 2, 29 h of screenshots provided by an Asian, young adult couple in a heterosexual relationship living in the United States.

⁸ N = 1, 24 h of screenshots provided by a white, young adult male living in the United States.

Additional information

Notes on contributors

Byron Reeves

Byron Reeves ([email protected], https://comm.stanford.edu/faculty-reeves) studies the psychological processing of media in the Department of Communication at Stanford University.

Nilam Ram

Nilam Ram ([email protected], https://hhd.psu.edu/contact/nilam-ram) studies how intensive longitudinal data contributes to knowledge of psychological processes in the Department of Human Development & Family Studies at Pennsylvania State University.

Thomas N. Robinson

Thomas N. Robinson ([email protected], https://profiles.stanford.edu/thomas-robinson) studies behavior change interventions to promote health and prevent disease in the Departments of Pediatrics and Medicine at Stanford University.

James J. Cummings

James J. Cummings ([email protected], http://www.bu.edu/com/profile/jim-cummings/) studies the psychological processing and effects of media in the College of Communication’s Division of Emerging Media Studies at Boston University.

C. Lee Giles

C. Lee Giles ([email protected], https://clgiles.ist.psu.edu) studies information and knowledge extraction, data mining, and artificial intelligence in the College of Information Science & Technology at Pennsylvania State University.

Jennifer Pan

Jennifer Pan ([email protected], http://jenpan.com/) studies political communication in authoritarian countries and computational methods in the Department of Communication at Stanford University.

Agnese Chiatti

Agnese Chiatti ([email protected]) studies knowledge extraction, retrieval and discovery in the Department of Information Sciences and Technology at Pennsylvania State University.

Mj Cho

MJ Cho ([email protected], http://comm.stanford.edu/doctoral-cho/) studies people’s use of media technologies and social behavior in the Department of Communication at Stanford University.

Katie Roehrick

Katie Roehrick ([email protected], https://comm.stanford.edu/doctoral-roehrick) uses computational linguistic analyses to study human-computer interaction and digital media in the Department of Communication at Stanford University.

Xiao Yang

Xiao Yang ([email protected], https://quantdev.ssri.psu.edu/people/xfy5031) studies quantitative methods and psychological dynamics in the Department of Human Development & Family Studies at Pennsylvania State University.

Anupriya Gagneja

Anupriya Gagneja ([email protected]) is a Software Engineer at Apple Inc, working with the Machine Learning Platform Team.

Miriam Brinberg

Miriam Brinberg ([email protected], https://quantdev.ssri.psu.edu/people/mjb6504) develops methods to examine interpersonal dynamics in the Department of Human Development & Family Studies at Pennsylvania State University.

Daniel Muise

Daniel Muise ([email protected]) studies the effects of Internet capacity on political information consumption & diffusion, trust, and development in the Department of Communication at Stanford University.

Yingdan Lu

Yingdan Lu ([email protected]) studies political communication and social effects of digital media in the Chinese context in the Department of Communication at Stanford University.

Mufan Luo

Mufan Luo ([email protected], https://comm.stanford.edu/doctoral-luo/) studies communication technology and psychological well-being, and is a PhD student in the Department of Communication at Stanford University.

Andrew Fitzgerald

Andrew Fitzgerald ([email protected]) studies the sociocultural and political economic impacts of digital media with a particular focus on national security and democracy in the Department of Communication at Stanford University.

Leo Yeykelis

Leo Yeykelis ([email protected], www.yeyleo.com) studies how psychological processes and HCI affect product design, and he works in the self-driving car, consumer and enterprise application spaces.