Pilot Instructor Rater Training: The Utility of the Multifacet Item Response Theory Model

Abstract

A multifacet 1-parameter item response theory (i.e., Rasch) model was used to examine interrater reliability training for pilot instructors. This model provides a means for examining individual instructor leniency or severity in ratings, difficulty of grade-sheet items, skill levels of flight crews, and interactions among these components. It was found that pilot instructor trainees differed in their levels of rating severity, and that higher crew resource management scores were easier to achieve than technical scores. Interaction analyses identified several pilot instructors who were evaluating crews in an unexpected manner, which is useful when providing feedback during training.