Beyond Self-Reports: Using Eye Tracking to Measure Topic and Style Differences in Attention to Social Media Content

ABSTRACT

Research on social media content overwhelmingly relies on self-reports, which we suggest are meaningfully limited and likely biased. Instead, we apply an under-utilized method—corneal eye tracking—for gauging attention to content in social media. We expose subjects to different types of Facebook content and track their gaze as they browse through posts. Substantively, we find that news and social content garner equal attention, with politics trailing behind both. We also find that the style of the post matters for attention patterns, with richer content (e.g., pictures, links) enhancing attention especially for social and news posts. Methodologically, we conclude that participants are unable to accurately report the topics and types of content available on the Facebook feed, even immediately after exposure. We discuss the implications of these findings and also make recommendations for appropriate methods in this area.

Acknowledgment

A special thank you to Anne-Bennett Smithson for her help with stimuli creation for this project.

Notes

¹ While this is not an exhaustive list of the topics considered by researchers, these three content areas—political, news, and social—are some of the most common topics that occur in research on social media content (see, for instance, Anderson, 2015; Barthel, Shearer, Gottfried, & Mitchell, 2015; Bode et al., 2014; Ellison et al., 2011; Gil de Zúñiga et al., 2012; Lenhart, 2015; Weeks & Holbert, 2013; among others).

² While our sample skews Democratic, this breakdown of party affiliation resembles the party affiliation of young adults ages 18–29 nationally. According to Jones (2014) in 2013, 53% of American youth were Democrats, 35% were Republicans, and 12% were Independents.

³ This instruction was given to ensure that participants spent adequate time on the task and to avoid click-through of the experiment. However, lab assistants were instructed not to interfere with participants, even if they were progressing through the study at faster than 10 sec per page. Moreover, post-test analysis suggests participants spent on average 9½ minutes with the task, with only 5 participants spending less than 10 sec per page. Therefore, we feel confident this instruction did not harm the validity of the study.

⁴ A series of independent-sample t-tests confirms there are no significant differences in attention between the two versions of the eye tracking task for any of the eight types of post examined in this study.

⁵ In this study, we examine political posts together (neutral, favoring Democrats, favoring Republicans). Additional analyses were performed to ensure no major attentional differences exist between these types of political posts, and the results confirmed no significant pattern of differences. Please contact the authors for details on these analyses.

⁶ This experimental design is not fully crossed, as the combination of picture posts for political topics lacked external validity. Therefore, news and social posts were fully crossed with all three post styles (pictures, external links, and status-only), but political posts were crossed only with two post styles (external links and status-only). We did not create picture-only posts for political content for two reasons: (1) we did not think this would be a common type of content on Facebook and (2) we would not be able to create enough variation in political picture posts (e.g., which would require 30 political picture posts for parity with the other conditions). Therefore, we argue that including political picture posts would harm the validity of our study and thus excluded them from our design.

⁷ Before performing these analyses, we examined the data for potential outliers. We tested for outliers in two ways: (1) in terms of posts (e.g., posts that received substantially more attention than other posts of that category) and (2) in terms of individuals (e.g., individuals who spent more time with a type of post than other individuals). We found no evidence of consistent outliers, which we defined as posts or individuals who spent more than 3 standard deviations different from their comparison group, as suggested for small sample sizes (Markus, 2012).

⁸ For status and picture posts, the number of characters in the post itself is controlled for. For link posts, character count also includes the total number of characters in the link title. This control is necessary because characters will draw attention regardless of the content. Therefore, all numbers reported in text to test our hypotheses are marginal means controlling for character count, as a more rigorous test of our hypotheses. Therefore, these means are slightly different from the descriptive statistics reported in Table 1, which do not control for number of characters in a post.

⁹ Additional analyses using the proportion of time spent on an average post, rather than the absolute time, largely confirm the results presented here. This proportion variable thus controls for interest in the task and reading speed, but as the output is less interpretable on its face, it is not reported in this article.

¹⁰ We again examine the data for outliers. In this case, we found one participant was an outlier (more than three standard deviations from the average score) for two of the three post topics, reporting that 77% of posts were social and 0% were political. This participant was this excluded from all analyses regarding exposure, as we suspect they were either not paying attention to the task or had a meaningful difference in their definition of post topics, leaving an N of 52. Exclusion of this outlier did not substantially affect the results. Please contact the lead author for more details.

¹¹ Additional analyses was performed at the request of the reviewer to test whether individual differences in political orientations (e.g., political interest, partisanship) may contribute to misreporting the frequency of content categories on Facebook. The results showed no support for this explanation: a series of t-tests found little evidence that political interest, partisanship, or attention to diverse Facebook topics or styles were related to estimates of the proportion of content viewed on the simulated feed. Please contact the lead author for details of these analyses.

Additional information

Funding

Funding for this paper was provided by George Mason University.