Initial validation of the trust of automated systems test (TOAST)

ABSTRACT

Trust is a key determinant of whether people rely on automated systems in the military and the public. However, there is currently no standard for measuring trust in automated systems. In the present studies, we propose a scale to measure trust in automated systems that is grounded in current research and theory on trust formation, which we refer to as the Trust in Automated Systems Test (TOAST). We evaluated both the reliability of the scale structure and criterion validity using independent, military-affiliated and civilian samples. In both studies we found that the TOAST exhibited a two-factor structure, measuring system understanding and performance (respectively), and that factor scores significantly predicted scores on theoretically related constructs demonstrating clear criterion validity. We discuss the implications of our findings for advancing the empirical literature and in improving interface design.

KEYWORDS:

• Trust in Automation
• Human-Machine Trust
• Trust Scale
• Confirmatory Factor Analysis

Data Availability Statement

The data described in this article are openly available in the Open Science Framework at https://osf.io/t25af.

Open Scholarship

This article has earned the Center for Open Science badge for Open Data and Open Materials. The materials and datas are openly accessible at https://osf.io/t25af.

Disclosure Statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Heather M. Wojton

Heather M. Wojton is a Research Staff Member at the Institute for Defense Analyses where she leads an interdisciplinary team focused on facilitating data-driven decision-making within national security organizations by advancing and applying statistical and behavioral science methodologies to evaluate military programs. She received her Ph.D. in Experimental Psychology from the University of Toledo.

Daniel Porter

Daniel Porter received his PhD in Cognitive Neuroscience from the University of Michigan in 2017. Since then, he has been a member of the research staff at the Institute for Defense Analyses working on test methods for artificial intelligence, autonomous systems, and human-systems integration.

Stephanie T. Lane

Stephanie T. Lane is an adjunct Research Staff Member at the Institute for Defense Analyses and a Senior Data Scientist at Netflix. She received her Ph.D. in Quantitative Psychology at the University of North Carolina at Chapel Hill, with a concentration in Biostatistics from the Gillings School of Global Public Health.

Chad Bieber

Chad Bieber is a Senior Research Engineer at the Johns Hopkins Applied Physics Lab where he runs test and evaluation programs on AI systems. Previously, he worked as a Research Staff Member at the Institute for Defense Analyses. A former pilot in the US Air Force, Chad received his Ph.D. in Aerospace Engineering from North Carolina State University.

Poornima Madhavan

Poornima Madhavan is a Principal Behavioral Scientist at MITRE, where her work involves the study of behavioral decision making and human-systems integration issues in cyber-physical systems. Previously, she worked as a Research Staff Member at the Institute for Defense Analyses. Poornima received her Ph.D. in Engineering Psychology from the University of Illinois at Urbana-Champaign and completed her post-doctoral fellowship at Carnegie Mellon University.