Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data

ABSTRACT

Graphics play a crucial role in statistical analysis and data mining. Being able to quantify structure in data that is visible in plots, and how people read the structure from plots is an ongoing challenge. The lineup protocol provides a formal framework for data plots, making inference possible. The data plot is treated like a test statistic, and lineup protocol acts like a comparison with the sampling distribution of the nulls. This article describes metrics for describing structure in data plots and evaluates them in relation to the choices that human readers made during several large Amazon Turk studies using lineups. The metrics that were more specific to the plot types tended to better match subject choices, than generic metrics. The process that we followed to evaluate metrics will be useful for general development of numerically measuring structure in plots, and also in future experiments on lineups for choosing blocks of pictures. Supplementary materials for this article are available online.

KEYWORDS:

• Cognitive perception
• Data mining
• Data science
• Data visualization
• Distance metrics
• Exploratory data analysis
• Information visualization
• Statistical graphics
• Visual inference

Acknowledgments

All plots are done with the ggplot2 (Wickham 2009) package in R. The document is written in knitr (Xie 2015). This article was submitted prior to Dianne Cook becoming Editor of JCGS, and was handled by the previous Editor.

Additional information

Funding

National Science Foundation [1007697].