Inter-rater reliability of query/probe-based techniques for measuring situation awareness

REFERENCES

• Bernardin, H. J., and M. R.Buckley. 1981. “Strategies in Rater Training.” The Academy of Management Review6: 205–212.
   Google Scholar
• Boring, R. L.2003. “Improving Human Scaling Reliability.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting47: 1820–1824.
   Google Scholar
• Boring, R. L., and R. L.West. 2008. “Constrained scaling in psychometric magnitude mapping.” Proceedings of the 24th Annual Meeting of the International Society for Psychophysics24: 297–302.
   Google Scholar
• Cohen, J.1960. “The coefficient of agreement for nominal scales.” Educational and Psychological Measurement20: 37–46.
   Web of Science ®Google Scholar
• Conway, J. M., and A. I.Huffcutt. 1997. “Psychometric Properties of Multisource Performance Ratings: A Meta-Analysis of Subordinate, Supervisor, Peer, and Self-Ratings.” Human Performance10: 331–360.
   Web of Science ®Google Scholar
• Conway, J. M., R. A.Jako, and D. F.Goodman. 1995. “A Meta-Analysis of Interrater and Internal Consistency Reliability of Selection Interviews.” Journal of Applied Psychology80: 565–579.
   Web of Science ®Google Scholar
• Devitt, J., M.Kurrek, M.Cohen, K.Fish, P.Fish, P.Murphy, and J.-P.Szalai. 1997. “Testing the Raters: Inter-Rater Reliability of Standardized Anaesthesia Simulator Performance.” Canadian Journal of Anesthesia [Journal canadien d'anesthésie]44: 924–928.
   PubMed Web of Science ®Google Scholar
• Drøivoldsmo, A., G.SkraaningJr., M.Sverrbo, J.Dalen, T.Grimstad, and G.Andresen. 1998. Continuous Measures of Situation Awareness and Workload. Halden: OECD Halden Reactor Project.
   Google Scholar
• Durso, F. T., M. K.Bleckley, and A. R.Dattel. 2006. “Does Situation Awareness Add to the Validity of Cognitive Tests?” Human Factors48: 721–733.
   PubMed Web of Science ®Google Scholar
• Durso, F. T., T. R.Truitt, C. A.Hackworth, J. M.Crutchfield, and C. A.Manning. 1998. “En Route Operational Errors and Situational Awareness.” The International Journal of Aviation Psychology8: 177–194.
   Web of Science ®Google Scholar
• Eckes, T.2008. “Rater Types in Writing Performance Assessments: A Classification Approach to Rater Variability.” Language Testing25: 155–185.
   Web of Science ®Google Scholar
• Endsley, M. R.1988. “Situation Awareness Global Assessment Technique (SAGAT).” In National Aerospace, Electronics Conference (NAECON), 789–795. New York, NY: IEEE.
   Google Scholar
• Endsley, M. R.1995. “Measurement of Situation Awareness in Dynamic Systems.” Human Factors37: 65–84.
   Web of Science ®Google Scholar
• Endsley, M. R.2000. “Direct Measurement of Situation Awareness: Validity and Use of SAGAT.” In Situation Awareness: Analysis and Measurement, edited by M. R.Endsley and D. J.Garland, 147–174. Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Fleiss, J. L., and J.Cohen. 1973. “The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability.” Educational and Psychological Measurement33: 613–619.
   Web of Science ®Google Scholar
• Fracker, M. L.1991. Measures of Situation Awareness Review and Future Directions. Dayton, OH: Wright-Patterson Air Force Base.
   Google Scholar
• Gatsoulis, Y., V. S.Gurvinder, and A. A.Dehghani-Sanij. 2010. “On the Measurement of Situation Awareness for Effective Human-Robot Interaction in Teleoperated Systems.” Journal of Cognitive Engineering and Decision Making4: 69–98.
   Google Scholar
• Haddock, G., J.Mccarron, N.Tarrier, and E. B.Faragher. 1999. “Scales to Measure Dimensions of Hallucinations and Delusions: The Psychotic Symptom Rating Scales (PSYRATS).” Psychological Medicine29: 879–889.
   PubMed Web of Science ®Google Scholar
• Hauss, Y., and K.Eyferth. 2003. “Securing Future ATM-Concepts’ Safety by Measuring Situation Awareness in ATC.” Aerospace Science and Technology7: 417–427.
   Web of Science ®Google Scholar
• Hogg, D. N., K.Follesø, F. S.Volden, and B.Torralba. 1995. “Development of a Situation Awareness Measure to Evaluate Advanced Alarm Systems in Nuclear Power Plant Control Rooms.” Ergonomics38: 2394–2413.
   Web of Science ®Google Scholar
• Howell, D.2002. Statistical Methods for Psychology. Pacific Grove, CA: Duxbury/Thomson Learning.
   Google Scholar
• Jeannot, E.2000. Situation Awareness, Synthesis of Literature Research. Brussels: Eurocontrol.
   Google Scholar
• Jeannot, E., C.Kelly, and D.Thompson. 2003. The Development of Situation Awareness Measures in ATM Systems. Brussels: Eurocontrol.
   Google Scholar
• Jones, D. G., and M. R.Endsley. 2004. “Use of Real-Time Probes for Measuring Situation Awareness.” The International Journal of Aviation Psychology14: 343–367.
   Web of Science ®Google Scholar
• Jonsson, A., and G.Svingby. 2007. “The Use of Scoring Rubrics: Reliability, Validity and Educational Consequences.” Educational Research Review2: 130–144.
   Google Scholar
• Karlsson, T., H.Jokstad, B. D.Meyer, C.Nilhlwing, S.Norrman, E. K.Puska, P.Raussi, and O.Tiihonen. 2001. OECD Halden Reactor Project: The HAMBO BWR Simulator of HAMMLAB. Halden Institutt for Energiteknikk.
   Google Scholar
• Kraemer, H. C.1992. “How Many Raters? Toward the Most Reliable Diagnostic Consensus.” Statistics in Medicine11: 317–331.
   PubMed Web of Science ®Google Scholar
• Landis, J. R., and G. G.Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics33: 159–174.
   PubMed Web of Science ®Google Scholar
• Lau, N., G. A.Jamieson, and G.SkraaningJr.2012. “Inter-Rater Reliability of Expert-Based Performance Measures.” In Proceedings of the 8th American Nuclear Society International Topical Meeting on Nuclear Plant Instrumentation, Control and Human-Machine Interface Technologies (NPIC & HMIT), 1974–1982. San Diego, CA: American Nuclear Society.
   Google Scholar
• Lievens, F.2001. “Assessor Training Strategies and Their Effects on Accuracy, Interrater Reliability, and Discriminant Validity.” Journal of Applied Psychology86: 255–264.
   PubMed Web of Science ®Google Scholar
• Maxwell, S. E., and H. D.Delaney. 1990. Designing Experiments and Analyzing Data: A Model Comparison Perspective. Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Mcguinness, B.2004. “Quantitative Analysis of Situational Awareness (QUASA): Applying Signal Detection Theory to True/False Probes and Self-Ratings.” Proceedings of the Ninth International Command and Control Research and Technology Symposium, San Diego, CA.
   Google Scholar
• Miller, T. J., T. H.Mcglashan, J. L.Rosen, K.Cadenhead, J.Ventura, W.Mcfarlane, D. O.Perkins, G. D.Pearlson, and S. W.Woods. 2003. “Prodromal Assessment With the Structured Interview for Prodromal Syndromes and the Scale of Prodromal Symptoms: Predictive Validity, Interrater Reliability, and Training to Reliability.” Schizophrenia Bulletin29: 703–715.
   PubMed Web of Science ®Google Scholar
• Mitchell, S. K.1979. “Interobserver Agreement, Reliability, and Generalizability of Data Collected in Observational Studies.” Psychological Bulletin86: 376–390.
   Web of Science ®Google Scholar
• Mumaw, R. J., E. M.Roth, K. J.Vicente, and C. M.Burns. 2000. “There is More to Monitoring a Nuclear Power Plant than Meets the Eye.” Human Factors42: 36–55.
   PubMed Web of Science ®Google Scholar
• Murphy, K. R., and C. O.Davidshofer. 1998. Psychological Testing: Principles and Applications. Upper Saddle River, NJ: Prentice Hall.
   Google Scholar
• Neal, A., M. A. Griffin, J. Paterson, and P. Bordia. 1998. “Development of Measures of Situation Awareness, Task Performance, and Contextual Performance in Air Traffic Control.” Fourth Australian Aviation Psychology Symposium, Sydney, Australia.
   Google Scholar
• Øwre, F., J.Kvalem, T.Karlsson, and C.Nihlwing. 2002. “A New Integrated BWR Supervision and Control System.” In Proceedings of IEEE 7th Conference on Human Factors and Power Plants, 4-41–4-47, Scottsdale, AZ.
   Google Scholar
• Patrick, J., N.James, A.Ahmed, and P.Halliday. 2006. “Observational Assessment of Situation Awareness, Team Differences and Training Implications.” Ergonomics49: 393–417.
   PubMed Web of Science ®Google Scholar
• Pew, R. W.2000. “The State of Situation Awareness Measurement: Heading Toward the Next Century.” In Situation Awareness Analysis and Measurement, edited by M. R.Endsley and D. J.Garland, 33–37. Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Prince, C., E.Ellis, M. T.Brannick, and E.Salas. 2007. “Measurement of Team Situation Awareness in Low Experience Level Aviators.” The International Journal of Aviation Psychology17: 41–57.
   Web of Science ®Google Scholar
• Ramos, K. D., S.Schafer, and S. M.Tracz. 2003. “Validation of the Fresno Test of Competence in Evidence Based Medicine.” BMJ326: 319–321.
   PubMed Web of Science ®Google Scholar
• Rosenzweig, S., T. P.Brigham, R. D.Snyder, G.Xu, and A. J.Mcdonald. 1999. “Assessing Emergency Medicine Resident Communication Skills Using Videotaped Patient Encounters: Gaps in Inter-Rater Reliability.” The Journal of Emergency Medicine17: 355–361.
   PubMed Web of Science ®Google Scholar
• Rousseau, R., S.Tremblay, and R.Breton. 2004. “Defining and Modeling Situation Awareness: A Critical Review.” In A Cognitive Approach to Situation Awareness: Theory and Application, edited by S. P.Banbury, and S.Tremblay, 3–21. Hampshire: Ashgate.
   Google Scholar
• Salmon, P. M., N. A.Stanton, G. H.Walker, and D.Green. 2006. “Situation Awareness Measurement: A Review of Applicability for C4i Environments.” Applied Ergonomics37: 225–338.
   PubMed Web of Science ®Google Scholar
• Salmon, P. M., N. A.Stanton, G. H.Walker, D.Jenkins, D.Ladva, L.Rafferty, and M.Young. 2009. “Measuring Situation Awareness in Complex Systems: Comparison of Measures Study.” International Journal of Industrial Ergonomics39: 490–500.
   Web of Science ®Google Scholar
• Saxton, E., S.Belanger, and W.Becker. 2012. “The Critical Thinking Analytic Rubric (CTAR): Investigating Intra-Rater and Inter-Rater Reliability of a Scoring Mechanism for Critical Thinking Performance Assessments.” Assessing Writing17: 251–270.
   Google Scholar
• Shrout, P. E.1998. “Measurement Reliability and Agreement in Psychiatry.” Statistical Methods in Medical Research7: 301–317.
   PubMedGoogle Scholar
• Shrout, P. E., and J. L.Fleiss. 1979. “Intraclass Correlations: Uses in Assessing Rater Reliability.” Psychological Bulletin86: 420–428.
   PubMed Web of Science ®Google Scholar
• Shrout, P. E., and S. P.Lane. 2012. “Reliability.” In APA Handbook of Research Methods in Psychology. Vol 1: Foundations, Planning, Measures, and Psychometrics, edited by H.Cooper, P. M.Camic, D. L.Long, A. T.Panter, D.Rindskopf, and K. J.Sher, 643–660. Washington, DC: American Psychological Association.
   Google Scholar
• Sims, J., and C. C.Wright. 2005. “The Kappa Statistics in Reliability Studies: Use, Interpretation, and Sample Size Requirements.” Physical Therapy85: 257–268.
   PubMed Web of Science ®Google Scholar
• Skraaning, G., Jr., M. H. R.Eitrheim, N.Lau, C.Nihlwing, L.Hurlen, and T.Karlsson. 2009. Coping with Automation in Future Plants: Results from the 2009 HAMMLAB Experiment. Halden: OECD Halden Reactor Project.
   Google Scholar
• Stanton, N. A.2010. “Situation Awareness: Where Have We Been, Where Are We Now and Where Are We Going?” Theoretical Issues in Ergonomics Science11: 1–6.
   Google Scholar
• Stanton, N. A., and M. S.Young. 1999. “What Price Ergonomics?” Nature399: 197–198.
   Web of Science ®Google Scholar
• Stanton, N. A., and M. S.Young. 2003. “Giving Ergonomics Away? The Application of Ergonomics Methods by Novices.” Applied Ergonomics34: 479–490.
   PubMed Web of Science ®Google Scholar
• Stemler, S. E.2004. “A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability.” Practical Assessment, Research & Evaluation. Accessed October 10, 2012. http://PAREonline.net/getvn.asp?v=9&n=4.
   Google Scholar
• Stuhlmann, J., C.Daniel, A.Dellinger, R.Kenton, and T.Powers. 1999. “A Generalizability Study of the Effects of Training on Teachers’ Abilities to Rate Children's Writing Using a Rubric.” Reading Psychology20: 107–127.
   Google Scholar
• Taylor, R. M.1990. “Situation Awareness Rating Technique (SART): The Development of a Tool for Aircrew Systems Design.” In Situational Awareness in Aerospace Operations. Neuilly Sur Seine: NATO-AGARD.
   Google Scholar
• Vidulich, M. A., and E. R.Hughes. 1991. “Testing a Subjective Metric of Situation Awareness.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting35: 1307–1311.
   Google Scholar
• Viswesvaran, C., D. S.Ones, and F. L.Schmidt. 1996. “Comparative Analysis of the Reliability of Job Performance Ratings.” Journal of Applied Psychology81: 557–574.
   Web of Science ®Google Scholar
• Waag, W. L., and M. R.Houck. 1994. “Tools for Assessing Situational Awareness in an Operational Fighter Environment.” Aviation, Space, and Environmental Medicine65: A13–A19.
   PubMedGoogle Scholar
• Willems, B. F., and M. Heiney. 2001. “Real-Time Assessment of Situation Awareness of Air Traffic Control Specialists on Operational Host Computer System and Display System Replacement Hardware.” 4th USA/Europe Seminars on Air Traffic Management Research and Development, Santa Fe, NM, USA. Eurocontrol [no pagination].
   Google Scholar
• Willems, B. F., and M.Heiney. 2002. Decision Support Automation Research in the En Route Air Traffic Control Environment. Atlantic City International Airport, NJ: FAA William J. Hughes Technical Center.
   Google Scholar