Construction and validation of an updated perfect automation schema (uPAS) scale

Abstract

The perfect automation schema is described as a representation people hold regarding the performance of automated systems, comprising initial high expectations for automated systems’ performance and low forgiveness after automated systems fail. Merritt, Unnerstall, Lee, and Huber have created a self-report measure of perfect automation schema comprising the two aforementioned factors, but this measure has demonstrated poor internal consistency estimates. In the present research, we created an updated perfect automation schema (uPAS) scale that showed acceptable reliability and validity estimates. In Study 1, we generated items that described both factors of perfect automation schema and conducted an exploratory factor analysis. In Study 2, we conducted a confirmatory factor analysis to confirm the uPAS scale composition and examined the scale’s convergent, discriminant, and criterion validity. We found acceptable reliability estimates for the new scale across both studies. In Study 2, however, we found the uPAS scale factors and the factors from Merritt and colleagues’ scale showed similar criterion validity across three trust-related criteria (trustworthiness perceptions, reliance intentions, and use endorsement). We conclude by offering a reliable uPAS scale to assess the perfect automation schema, which showed comparable criterion-related validity to Merritt and colleagues’ scale.

Keywords:

• automation schema
• reliability
• validity

Acknowledgements

Distribution A. Approved for public release; distribution unlimited. 88ABW-2020-3274; Cleared 22 Oct 2020. The views expressed are those of the authors and do not necessarily reflect the official policy or position of the Department of the Air Force, the Department of Defense, or the U.S. government. The research was approved by the Air Force Research Laboratory 711th Human Performance Wing Institutional Review Board [protocol # FWR20200082E].

Authors’ contributions

Anthony M. Gibson conceptualized the research idea and led the research design, co-led data analysis, and co-led writing the manuscript. August Capiola assisted in developing the research design and co-led writing the manuscript. Gene M. Alarcon assisted in developing the research design, co-led data analysis, and contributed to writing the manuscript. Michael A. Lee assisted in developing the research design, contributed to data analysis, and contributed to writing the manuscript. Sarah A. Jessup assisted in developing the research design and contributed to writing the manuscript. Izz aldin Hamdan assisted in developing the research design and contributed to writing the manuscript.

Data availability statement

The data that support the findings of this study are available from the corresponding author, AC, upon reasonable request.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 Decision support systems, automated aides, automated system, and automation are terms that are often used synonymously and interchangeably within various domains that study these systems. Throughout this paper, we focus on the term automated systems for continuity.

2 Our choice to conduct separate analyses to investigate the effects of manipulations on criteria for people with different levels of all-or-none thinking and high expectations separately allows us to tease apart the role of each facet in the trust process and aligns with Merritt et al.’s (2015) separate analyses of PAS facets with criteria of interest.

3 The conceptual separation of the factors composing PAS is not new. For a discussion on all-or-none thinking as a germane measure of PAS and high expectations as something closer to propensity to trust machines, see Merritt et al. 2015, pp. 750-751).

Additional information

Notes on contributors

Anthony M. Gibson

Anthony M. Gibson is a Personnel Psychologist at the United States Department of Veterans Affairs. At the time this work was conducted, he was a Consortium Post-Doctoral Research Fellow. He earned his PhD in Industrial/Organizational and Human Factors Psychology at Wright State University in 2019. His research interests are careless responding, counterproductive work behavior, and trust in automation.

August Capiola

August Capiola is a Research Psychologist in the Collaborative Teaming Section within the 711th Human Performance Wing at Wright-Patterson Air Force Base, Ohio. He earned his PhD in Human Factors and Industrial/Organizational Psychology at Wright State University. His research interests include human trust toward automation, robots, and autonomous systems, and swift trust in ad hoc teams.

Gene M. Alarcon

Gene M. Alarcon is a Senior Research Psychologist in the Collaborative Interfaces and Teaming Branch with the 711^th Human Performance Wing at Wright-Patterson Air Force Base, Ohio. He earned is PhD in Industrial/Organizational and Human Factors Psychology at Wright State University. His research interests include interpersonal trust, trust in automation/autonomy, and statistics.

Michael A. Lee

Michael A. Lee is a GDIT, Inc. Research Analyst in the Collaborative Teaming Section within the 711th Human Performance Wing at Wright-Patterson Air Force Base, Ohio. He earned his MA in Clinical Psychology at University of Dayton. His research interests include the trust process, psychometrics, scale development, data quality, and careless responding.

Sarah A. Jessup

Sarah A. Jessup is a Consortium Doctoral Research Fellow in the Collaborative Teaming Section within the 711th Human Performance Wing at Wright-Patterson Air Force Base, Ohio. She earned her MS in Human Factors and Industrial/Organizational Psychology at Wright State University. Her research interests include human-robot interaction, trust in automated agents, and social neuroscience.

Izz aldin Hamdan

Izz aldin Hamdan is a GDIT, Inc. Research Analyst in the Collaborative Teaming Section within the 711th Human Performance Wing at Wright-Patterson Air Force Base, Ohio. He earned his MPS in Industrial/Organizational Psychology at George Mason University. His research interests include human-robot interaction, human-swarm interaction, and swift trust in ad hoc teams.