Evaluating non-technical skills and mission essential competencies of pilots in military aviation environments

References

• Alinger, G. M., R. Beard, W. Bennett, C. M. Colegrove, and M. Garrity. 2007. Understanding Mission Essential Competencies as a Work Analysis Method. United States Air Force Research Laboratory Report AFRL-HE-AZ-TR-2007-0034, Mesa, AZ, August.
   Google Scholar
• Brown, R. D., and N. M. Hauenstein. 2005. “Inter-rater Agreement Reconsidered: An Alternative to the rwg Indices.” Organizational Research Methods 8: 165–184.10.1177/1094428105275376
   Web of Science ®Google Scholar
• Cannon-Bowers, J. A., and E. Salas, eds. 1998. Making Decisions Under Stress: Implications for Individual and Team Training. Washington, DC: American Psychological Association (APA).
   Google Scholar
• Crandall, B., G. Klein, and R. R. Hoffman. 2006. Working Minds: A Practitioner’s Guide to Cognitive Task Analysis. Cambridge, MA: MIT Press.
   Google Scholar
• Dedy, N. J., P. Szasz, M. Louridas, E. M. Bonrath, H. Husslein, and T. P. Grantcharov. 2015. “Objective Structured Assessment of Non-technical Skills: Reliability of a Global Rating Scale for the In-training Assessment in the Operating Room.” Surgery 157: 1002–1013.10.1016/j.surg.2014.12.023
   PubMed Web of Science ®Google Scholar
• Dreyfus, H., and S. Dreyfus. 1986. Mind Over the Machine: The Power of Human Intuition and Expertise in the Era of the Computer. New York: Free Press.
   Google Scholar
• Feltovich, P. J., R. R. Hoffman, D. Woods, and A. Roesler. 2004. “Keeping it Too Simple: How the Reductive Tendency Affects Cognitive Engineering.” IEEE Intelligent Systems 19 (3): 90–94.10.1109/MIS.2004.14
   Web of Science ®Google Scholar
• Flin, R., and L. Martin. 2001. “Behavioral Markers for Crew Resource Management: A Review of Current Practice.” The International Journal of Aviation Psychology 11 (1): 95–118.10.1207/S15327108IJAP1101_6
   Web of Science ®Google Scholar
• Flin, R., L. Martin, K. M. Goeters, H. J. Hörmann, R. Amalberti, C. Valot, and H. Nijhuis. 2003. “Development of the NOTECHS (Non-Technical Skills) System for Assessing Pilots’ CRM Skills.” Human Factors and Aerospace Safety 3 (2): 95–117.
   Google Scholar
• Goldsmith, T. E., and P. J. Johnson. 2002. “Assessing and Improving Evaluation of Aircrew Performance.” The International Journal of Aviation Psychology 12 (3): 223–240.10.1207/S15327108IJAP1203_3
   Web of Science ®Google Scholar
• Gontar, P., and H. J. Hoermann. 2015a. “Inter-rater Reliability at the Top End: Measures of Pilots’ Non-technical Performance.” The International Journal of Aviation Psychology 25: 171–190.10.1080/10508414.2015.1162636
   Web of Science ®Google Scholar
• Gontar, P., and H. J. Hoermann. 2015b. Reliability of Instructor Pilots’ Non-Technical Skills Ratings. In Proceedings of the 18th International Symposium on Aviation Psychology, 366–371. Dayton, OH.
   Google Scholar
• Gorman, J. C., N. J. Cooke, and J. L. Winner. 2006. “Measuring Team Situation Awareness in Decentralized Command and Control Environments.” Ergonomics 49: 1312–1325.10.1080/00140130600612788
   PubMed Web of Science ®Google Scholar
• Grote, G., M. Kolbe, E. Zala-Mezö, N. Bienefeld-Seall, and B. Künzle. 2010. “Adaptive Coordination and Heedfulness Make Better Cockpit Crews.” Ergonomics 53: 211–228.
   PubMed Web of Science ®Google Scholar
• Helmreich, R. L., J. R. Klinect, and J. A. Wilhelm. 1999. Models of Threat, Error, and CRM in Flight Operations. In Proceedings of the Tenth International Symposium on Aviation Psychology, 677–682. Columbus, OH: The Ohio State University.
   Google Scholar
• ICAO. 2002. Line Operations Safety Audit (LOSA). Doc 9803. AN/761. Montreal: International Civil Aviation Authority.
   Google Scholar
• Johnston, N. 1997. “Teaching Human Factors for Airline Pilots.” In Design Instruction for Human Factors Training in Aviation, edited by G. J. F. Hunt, 127–160. Aldershot: Ashgate.
   Google Scholar
• Johnston, J. A., K. A. Smith-Jentsch, and J. A. Cannon-Bowers. 1997. “Performance Measurement Tools for Enhancing Team Decision Making.” In Team Performance Assessment and Measurement: Theory, Methods and Applications, edited by M. T. Brannick, E. Salas and C. Prince, 311–327. Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Klinect, J. R., J. A. Wilhelm, and R. L. Helmreich. 1999. “Threat and Error Management: Data from Line Operations Safety Audits.” Proceedings of the 10th International Symposium on Aviation Psychology, 683–688. Columbus, OH: The Ohio State University.
   Google Scholar
• Kontogiannis, T., and S. Malakis. 2013. “Strategies in Coping with Complexity: Development of a Behavioral Marker System for Air Traffic Controllers.” Safety Science 57: 27–34.10.1016/j.ssci.2013.01.014
   Web of Science ®Google Scholar
• MacMillan, J., E. B. Entin, R. Morley, and Jr W. Bennett. 2006. “Measuring Team Performance in Complex and Dynamic Military Environments: The SPOTLITE Method.” Military Psychology 25: 266–279.
   Web of Science ®Google Scholar
• Matthews, G., J. S. Warm, T. H. Shaw, and V. S. Finomore. 2014. “Predicting Battlefield Vigilance: A Multivariate Approach to Assessment of Attentional Resources.” Ergonomics 57: 856–875.10.1080/00140139.2014.899630
   PubMed Web of Science ®Google Scholar
• Mavin, T., and G. Dall’ Alba. 2010. “A Model for Integrating Technical Skills and NTS in Assessing Pilot’s Performance.” In 9th International Symposium of Australian Aviation Psychology, Sydney, Australia.
   Google Scholar
• Negard, V., O. E. Hatlevik, M. Martinussen, and A. Lervag. 2011. “An Airman’s Personal Attitude: Pilots’ Point of View.” Aviation 15 (4): 101–111.10.3846/16487788.2011.651789
   Google Scholar
• O’Connor, P., H. J. Hörmann, R. Flin, M. Lodge, and K. M. Goeters. 2002. “Developing a Method for Evaluating Crew Resource Management Skills: A European Perspective.” The International Journal of Aviation Psychology 12 (3): 263–285.10.1207/S15327108IJAP1203_5
   Web of Science ®Google Scholar
• van der Pal, J., E. J. Boland, and M. de Rivecourt. 2009. Competency-based Design of F-16 Qualification Training. National Aerospace Laboratory NLR Report NLR-TP-2009-373, Amsterdam.
   Google Scholar
• Riley, J. M., M. R. Endsley, C. A. Bolstad, and H. M. Cuevas. 2006. “Collaborative Planning and Situation Awareness in Army Command and Control.” Ergonomics. 49: 1139–1153.10.1080/00140130600612614
   PubMed Web of Science ®Google Scholar
• Ruff-Stahl, H. J., D. Vogel, N. Dmoch, A. Krause, A. Strobl, D. Farsch, and R. Stehr. 2016. “Measuring CRM Aptitude: Is NOTECHS a Suitable Tool for Pilot Selection?” International Journal of Aviation, Aeronautics, and Aerospace 3 (3), 4. doi:10.15394/ijaaa.2016.1128.
   Web of Science ®Google Scholar
• Salas, E., C. Shawn Burke, C. A. Bowers, and K. A. Wilson. 2001. “Team Training in the Skies: Does Crew Resource Management (CRM) Training Work?” Human Factors: The Journal of the Human Factors and Ergonomics Society 43 (4): 641–674.10.1518/001872001775870386
   PubMed Web of Science ®Google Scholar
• Seamster, T. L., D. A. Boehm-Davis, R. W. Holt, and K. Schult. 1998. Developing Advanced Crew Resource Management (ACRM) Training: A Training Manual. Washinton, DC: Federal Aviation Administration.
   Google Scholar
• Symons, S., M. France, J. Bell, and W. Bennett Jr. 2006. Linking Knowledge and Skills to Mission Essential Competency-based Syllabus: Development for Distributed Mission Operations. United States Air Force Research Laboratory Report AFRL-HE-AZ-TR-2006-0041, Mesa, AZ, July.
   Google Scholar
• Thomas, M. 2004. “Predictors of Threat and Error Management: Identification of Core Non-technical Skills and Implications for Training System Design.” The International Journal of Aviation Psychology 14 (2): 207–231.10.1207/s15327108ijap1402_6
   Web of Science ®Google Scholar
• Tossell, C. D., E. Wiese, M. J. Garrity, T. Denning, and G. M. Alleger. 2006. “Developing Command and Control Performance-based Training Criteria in a Network Centric Environment.” 11th International Command & Control Research & Technology Symposium, International Command & Control Institute, Cambridge, UK.
   Google Scholar
• Weber, D. E., W. M. Roth, T. J. Mavin, and S. W. Dekker. 2013. “Should We Pursue Inter-rater Reliability or Diversity?: An Empirical Study of Pilot Performance Assessment.” Aviation in Focus: Journal of Aeronautical Sciences 4 (2): 34–58.
   Google Scholar
• Weber, D. E., T. J. Mavin, W. M. Roth, E. Henriqson, and S. W. Dekker. 2014. “Exploring the Use of Categories in the Assessment of Airline Pilots’ Performance as a Potential Source of Examiners’ Disagreement.” Journal of Cognitive Engineering and Decision Making 8: 248–264.10.1177/1555343414532813
   Google Scholar
• Yule, S., D. Rowley, R. Flin, N. Maran, G. Youngson, J. Duncan, and S. Paterson-Brown. 2009. “Experience Matters: Comparing Novice and Expert Ratings of Non-technical Skills Using the NOTSS System.” ANZ Journal of Surgery 79: 154–160.10.1111/ans.2009.79.issue-3
   PubMed Web of Science ®Google Scholar