Examining external validity issues in research with human operation of unmanned vehicles

References

• Boles , DB and Adair , LP . 2001a . The multiple resources questionnaire (MRQ) . In: Proceedings of the 45th annual meeting of the human factors and ergonomics society . 2001a , 8–12 October 2001 , Minneapolis. MN. Santa Monica, CA: HFES, 1790–1794
   Google Scholar
• Boles , DB and Adair , LP . 2001b . Validity of the multiple resources questionnaire (MRQ) . In: Proceedings of the 45th annual meeting of the human factors and ergonomics society . 2001b , 8–12 October 2001 , Minneapolis. MN. Santa Monica, CA: HFES, 1795–1799
   Google Scholar
• Boles , DB . 2007 . Predicting dual-task performance with the multiple resources questionnaire (MRQ) . Human Factors , 49 : 32 – 45 .
   PubMed Web of Science ®Google Scholar
• Burke , J and Murphy , RR . 2004. Human–robot interaction in USAR technical search: two heads are better than one. In: 13th IEEE international workshop on robot and human interactive communication, 20–22 September 2004, Kurashiki, Okayama, Japan. Piscataway, NJ: IEEE, 13, 307–312
   Google Scholar
• Burke , J and Murphy , RR . 2007 . RSVP: an investigation of remote shared visual presence as common ground for human–robot teams . In: Proceedings of the 2nd ACM conference on human robot interaction . 2007 , 8–10 March 2007 , Arlington. VA. New York, NY: ACM, 161–168
   Google Scholar
• Burke , JL . 2004 . Moonlight in Miami: a field study of human–robot interaction in the context of an urban search and rescue disaster response training exercise . Human–Computer Interaction , 19 : 85 – 116 .
   Web of Science ®Google Scholar
• Campbell , DT . 1957 . Factors relevant to the validity of experiments in social settings . Psychological Bulletin , 54 : 297 – 312 .
   PubMed Web of Science ®Google Scholar
• Campbell , DT and Stanley , J . 1963 . Experimental and quasi-experimental designs for research , Chicago , IL : Rand McNally .
   Google Scholar
• Casper , J and Murphy , RR . 2003 . Human–robot interactions during the robot-assisted urban search and rescue response at the World Trade Center . IEEE Transactions on Systems, Man and Cybernetics, Part B , 33 : 367 – 385 .
   PubMed Web of Science ®Google Scholar
• Chadwick , R . 2005 . The impacts of multiple robots and displays views: an urban search and rescue simulation . In: Proceedings of the 49th annual meeting of the human factors and ergonomics society . 2005 , 26–30 September 2005 , Orlando. FL. Santa Monica, CA: HFES, 49 (3), 387–391
   Google Scholar
• Chadwick , R . 2006 . Operating multiple semi-autonomous robots: monitoring, responding, & detecting . In: Proceedings of the 50th annual meeting of the human factors and ergonomics society . 2006 , 16–20 October 2006 , San Francisco. CA. Santa Monica, CA: HFES, 329–333
   Google Scholar
• Chen , J . 2010 . UAV-guided navigation for ground robot tele-operation in a military reconnaissance environment . Ergonomics , 53 ( 8 ) : 940 – 950 .
   PubMed Web of Science ®Google Scholar
• Chen , J , Barnes , M and Harper-Sciarini , M . 2011 . Supervisory control of multiple robots: human-performance issues and user-interface design . IEEE Transactions on Systems, Man, and Cybernetics-Part C: Application and Reviews , 41 ( 4 ) : 435 – 454 .
   PubMed Web of Science ®Google Scholar
• Chen , J and Terrence , P . 2009 . Effects of imperfect automation and individual differences on concurrent performance of military and robotics tasks in a simulated multitasking environment . Ergonomics , 52 ( 8 ) : 907 – 920 .
   PubMed Web of Science ®Google Scholar
• Cook , TD and Campbell , DT . 1979 . Quasi-experimentation: design and analysis issues for field settings , Hopewell , NJ : Houghton Mifflin .
   Google Scholar
• Cooke , N and Durso , F . 2008 . Stories of modern technology failures and cognitive engineering successes , Boca Raton , FL : Taylor & Francis Group .
   Google Scholar
• Cronbach , LJ . 1982 . Designing evaluations of educational and social programs , San Francisco , CA : Jossey-Bass .
   Google Scholar
• Driskell , JE and Salas , E . 1992 . “ Can you study real teams in contrived settings? The value of small group research to understanding teams ” . In Teams: their training and performance , Edited by: Sweezy , R and Salas , E . 101 – 124 . Norwood , NJ : Ablex .
   Google Scholar
• Endsley , M . 1995 . Toward a theory of situation awareness in dynamic systems . Human Factors , 37 ( 1 ) : 32 – 64 .
   Web of Science ®Google Scholar
• Endsley , M . 2000 . “ Direction measurement of situation awareness: validity and use of SAGAT ” . In Situation awareness and analysis and assessment , Edited by: Endsley , M and Garland , D . 147 – 173 . Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Evans , AW III . 2006 . “ Exploring human–robot interaction: emerging methodologies and environments ” . In Human factors of remotely piloted vehicles , Edited by: Cooke , NJ . 345 – 358 . Amsterdam , NL : Elsevier .
   Google Scholar
• Fatolitis , P . 2008. Initial validation of novel performance-based measures: mental rotation and psychomotor ability. Unpublished dissertation. University of Central Florida
   Google Scholar
• Fincannon , T . Curtis, M., and Jentsch, F., 2006. Familiarity and expertise in the recognition of vehicles from an unmanned ground vehicle. In: Proceedings of the 50th annual meeting of the human factors and ergonomics society, 16–20 October 2006, San Francisco, CA. Santa Monica, CA: HFES, 1218–1222
   Google Scholar
• Fincannon , T . et al., 2008. Target identification support and location support among teams of unmanned system operators [electronic version]. In: Proceedings of the 26th annual army science conference [online], 1–4 December 2008, Orlando, FL. Available from: http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA504210&Location=U2&doc=GetTRDoc.pdf [Accessed 1 August 2012]
   Google Scholar
• Fincannon , T . et al., 2009. The influence of team size and communication modality on team effectiveness with unmanned systems. In: Proceedings of the 53rd annual meeting of the human factors and ergonomics society, 19–23 October 2009, San Francisco, TX. Santa Monica, CA: HFES, 419–423
   Google Scholar
• Fincannon , T . et al., 2010a. Acquisition of skill sets and mental models over time. In: 3rd international conference on applied human factors and ergonomics, 17–20 July 2010, Miami, FL. New York, NY: CRC Press
   Google Scholar
• Fincannon , T . 1615 . et al , Santa Monica, CA : HFES . 2010b. Some good and bad with spatial ability in three person teams that operate multiple unmanned systems. In: Proceedings of the 54th annual meeting of the human factors and ergonomics society, 27 September–1 October 2010, San Francisco, CA–1619
   Google Scholar
• Fincannon , T . 2011 . Team size, team role, communication modality, and team coordination in the distributed operation of multiple heterogeneous unmanned vehicles . Journal of Cognitive Engineering and Decision Making , 5 ( 1 ) : 106 – 131 .
   Google Scholar
• Fiore , SM . 2008 . Power and promise: cognitive psychology and cognitive technology . Cognitive Technology , 13 ( 1 ) : 5 – 8 .
   Google Scholar
• Fiore , SM . in press . “ Cognition and technology: interdisciplinarity and the impact of cognitive engineering research on organizational productivity ” . In Oxford handbook of industrial and organizational psychology , Edited by: Kozlowski , S . New York , NY : Oxford University Press .
   Google Scholar
• Fiore , SM and Salas , E . 2007 . “ Problems and possibilities: strategically pursuing a science of learning in distributed environments ” . In Towards a science of distributed learning , Edited by: Fiore , SM and Salas , E . 237 – 264 . Washington , DC : American Psychological Association .
   Google Scholar
• Garcia , F and Wantchekon , L . 2010 . Theory, external validity, and experimental inference: some conjectures . The Annals of the American Academy of Political and Social Science , 628 : 132 – 147 .
   Web of Science ®Google Scholar
• Goodrich , M . 2008 . Supporting wilderness search and rescue using a camera-equipped mini UAV . Journal of Field Robotics , 25 ( 1–2 ) : 89 – 110 .
   Web of Science ®Google Scholar
• Hart , S and Staveland , L . 1988 . “ Development of NASA TLX (task load index): results of empirical and theoretical research ” . In Human mental workload , Edited by: Hancock , P and Meshkati , N . 139 – 183 . Amsterdam : Elsevier .
   Google Scholar
• Hegarty , M and Waller , D . 2005 . “ Individual differences in spatial ability ” . In The Cambridge handbook of visuospatial thinking , Edited by: Shah , P and Miyake , A . 121 – 169 . Cambridge : Cambridge University Press .
   Google Scholar
• Huthmann , A . 2009. Presence-dependent performance differences between virtual simulations and miniature worlds. Unpublished dissertation. University of Central Florida
   Google Scholar
• Jentsch , FG and Bowers , C . 1998 . Evidence for the validity of PC-based simulations in studying aircrew coordination . International Journal of Aviation Psychology , 8 ( 3 ) : 243 – 260 .
   Web of Science ®Google Scholar
• Jentsch , FG . 2010 . “ Model world: military HRI research conducted using a scale MOUT facility ” . In Human–robot interactions in future military operations , Edited by: Barnes , M and Jentsch , J . 419 – 431 . Aldershot , UK : Ashgate .
   Google Scholar
• Jones , D . 2000 . “ Subjective measures of situation awareness ” . In Situation awareness and analysis and assessment , Edited by: Endsley , M and Garland , D . 113 – 128 . Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Lucas , J . 2003 . Theory-testing, generalization, and the problem of external validity . Sociological Theory , 21 ( 3 ) : 236 – 253 .
   Web of Science ®Google Scholar
• Lynch , J . 1999 . Theory and external validity . Journal of the Academy of Marketing Science , 27 ( 3 ) : 367 – 376 .
   Web of Science ®Google Scholar
• McDermott , P . et al., 2005. Effective human to human communication of information provided by an unmanned vehicle. In: Proceedings of the 49th annual meeting of the human factors and ergonomics society, 26–30 September 2005, Orlando, FL. Santa Monica, CA: HFES, 402–406
   Google Scholar
• Murphy , RR and Burke , J . 2005 . Up from the rubble: lessons learned about HRI from search and rescue . In: Proceedings of the 49th annual meeting of the human factors and ergonomics society . 2005 , 26–30 September 2005 , Orlando. FL. Santa Monica, CA: HFES, 437–441
   Google Scholar
• Murphy , RR and Burke , JL . 2010 . “ The safe human–robot ratio ” . In Human–robot interactions in future military operations , Edited by: Barnes , M and Jentsch , F . 31 – 49 . Burlington , VT : Ashgate .
   Google Scholar
• Phillips , E . 1491 . et al , Santa Monica, CA : HFES . 2011. From tools to teammates: toward the development of appropriate mental models for intelligent robots. In: Proceedings of the 55th annual meeting of the human factors and ergonomics society, 19–23 September 2011, Las Vegas, NV–1495
   Google Scholar
• Rehfeld , S . 2006. The impact of mental transformation training across levels of automation on spatial awareness in human–robot interaction. Unpublished dissertation. University of Central Florida
   Google Scholar
• Rehfeld , S . et al., 2005. Collaborative teamwork with unmanned vehicles in military missions. In: Proceedings of the 11th international conference on human–computer interaction [electronic version], 22–27 July 2005, Las Vegas, NV
   Google Scholar
• Riley , J . 2010 . “ Situation awareness in human–robot interaction: challenges and user interface requirements ” . In Human–robot interactions in future military operations , Edited by: Barnes , M and Jentsch , J . 172 – 191 . Aldershot , UK : Ashgate .
   Google Scholar
• Schuster , D . et al., 2008. The role of spatial ability in the relationship between video game experience and route effectiveness among unmanned vehicle operators [electronic version]. In : Proceedings of the 26th annual army science conference [online], 1–4 December 2008, Orlando, FL. Available from: http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA505705&Location=U2&doc=GetTRDoc.pdf [Accessed 1 August 2012]
   Google Scholar
• Schuster , D . et al., 2011. A research approach to shared mental models and situation assessment in future robot teams. In: Proceedings of the 55th annual meeting of the human factors and ergonomics society, 19–23 September 2011, Las Vegas, NV. Thousand Oaks, CA: Sage, 55 (1), 456–460
   Google Scholar
• Sellers , B . 1 . Fincannon, T., and Jentsch, F., 2011. Indexing spatial ability across team size: the influence of the weakest link, strongest link, and aggregate ability on performance with multiple unmanned systems. In: Proceedings of the 55th annual meeting of the human Factors and ergonomics society, 19–23 September, Las Vegas, NV. Thousand Oaks, CA: Sage, 55 , 899–903
   Google Scholar
• Shadish , WR . 2010 . Combining estimates of effect size: the handbook of research synthesis and meta-analysis , New York , NY : Russel Sage .
   Google Scholar
• Shadish , WR , Cook , TD and Campbell , DT . 2002 . Experimental and quasi-experimental designs for generalized causal inference , New York , NY : Houghton Mifflin Company .
   Google Scholar
• Steinfeld , A . et al., 2006. Common metrics for human–robot interaction. In: Proceedings of the 1st ACM conference on human robot interaction, 2–3 March 2006, Salt Lake City, UT. New York, NY: ACM, 33–40
   Google Scholar
• Stokes , DE . 1997 . Pasteur's quadrant: basic science and technological innovation , Washington , DC : Brookings Institution .
   Google Scholar
• Taylor , R . 1990. Situation awareness rating techinique (SART): the development of a tool for aircrew systems design. In: Aerospace medical panel symposium, 2–6 October 1989, Copenhagen, Denmark. Neuilly Sur Seine, France: AGARD, 3, 1–17
   Google Scholar
• Wang , J and Lewis , M . 2007 . Human control for cooperating robot teams . In: Proceedings of the 2nd ACM conference on human robot interaction . 2007 , 8–10 March 2007 , Arlington. VA. New York, NY: ACM, 9–16
   Google Scholar
• Wheelan , S . 2009 . Group size, group development, and group productivity . Small Group Research , 40 ( 2 ) : 247 – 262 .
   Web of Science ®Google Scholar
• Woods , DD . 2004 . Envisioning human–robot coordination in future operations . IEEE Transactions on Systems, Man and Cybernetics, Part C , 34 ( 2 ) : 210 – 218 .
   PubMed Web of Science ®Google Scholar