The Effects of Tablet-Based Electronic Grading on Airline Evaluator Performance

ABSTRACT

Objective: This experimental study assessed the effects of an Apple iPad (Apple and iPad are trademarks of Apple Inc., registered in the U.S. and other countries) grading instrument on airline evaluator performance.

Background: Extant research exploring the effects of grading formats on evaluator performance focused on pen and paper modalities. This research investigated how a contemporary iPad-based grading format affected airline evaluator performance.

Method: Forty-five evaluators from a major U.S. airline graded a Boeing 767 (Boeing is a wordmark of The Boeing Company, registered in the U.S. and other countries) flight crew experiencing an engine failure during takeoff. Three grading formats were used: pen and paper, a paper-based equivalent of the iPad grading format designed to replicate the function of the iPad, and the iPad. The effects of the grading formats were measured using the following dependent variables: recorded technical criteria, recorded non-technical criteria, and correlation and inter-rater agreement with a referent score.

Results: Members of the iPad paper-based equivalent and iPad groups recorded significantly more technical and non-technical criteria compared to participants using pen and paper (p < .001). Members of the iPad paper-based equivalent and iPad groups had significantly higher correlation (p = .01) and inter-rater agreement levels (p < .01) compared to participants using pen and paper.

Conclusion: The results of the study failed to support the iPad as a superior grading format. Research outcomes suggested structured grading may be a more important factor in predicting airline evaluator performance than the type of grading format used.

Acknowledgments

I would like to recognize Dr. Terrence Kelly and the aviation faculty of Parks College of Engineering, Aviation and Technology at Saint Louis University for their inspiration and guidance. To the dozens of airline evaluators who volunteered their time to participate in this research, thank you for your efforts. Finally, I would like to thank the subject airline’s flight training management for their approval and willingness to grant access to the company’s resources.

Disclosure Statement

It is important to note the author of this article is a pilot and evaluator for the subject airline. While the subject airline provided access to their facilities and resources, research-related funding was not provided to the author by the subject airline. All participants volunteered their time without remuneration. To ensure strict adherence to research protocols, oversight was provided by Saint Louis University’s Institutional Review Board.

Data Availability Statement

The data that support the findings of this study are openly available via 4TU Centre for Research Data at https://doi.10.4121/uuid:0586027a-3027-479f-9e19-2c8e1bfcc082, reference number [10.4121/uuid:0586027a-3027-479f-9e19-2c8e1bfcc082].