Introducing a Subject Matter Expert–Based Utility Analysis Approach to Assessing the Utility of Organizational Interventions Such as Crew Resource Management Training

Abstract

Utility analysis is a series of statistical procedures used to quantify the fiscal gains accrued from implementing human resource management interventions (e.g., training programs) over their associated costs. Although empirical evaluation studies are typically used to generate the required utility parameters, particular circumstances might make it difficult if not impossible to conduct such studies. In response to situations where it is not possible to conduct formal empirical evaluation studies, we present a subject matter expert (SME)-based approach to implementing utility analysis-based evaluations using a crew resource management (CRM) case study in commercial aviation as an illustrative example. The SME-based approach was effective in generating utility estimates when traditional methods of utility analyses were not feasible.

ACKNOWLEDGMENT

This research was sponsored by a contract from the Federal Railroad Administration awarded to the Texas Transportation Institute, Texas A&M University, College Station, TX.

Notes

¹We acknowledge Patankar and Taylor's (2000) HFES proceedings on “making a business case for the human factors programs in aviation maintenance.” However, their approach is quite rudimentary and is far removed from a formal utility analysis approach as represented in the industrial/organizational psychology literature.

²In the semistructured interview form, SMEs were provided with definitions of U.S. air carrier, critical event, accident, incident, fatal injury, serious injury, and substantial damage.

³Although we were unable to locate any published archival or objective data that could be used to calculate the cost of CRM training per trainee (C), the SMEs who were currently working for an airline had access to these data. Observation of and follow-up questions about SMEs' judgments regarding the cost of CRM training per trainee revealed that when making their judgments, the SMEs relied heavily on these archival, airline-specific data.

⁴The descriptors of none, small, moderate, large, and very large effects corresponded to ds of 0, .20, .50, .80, and 1.10, respectively (e.g., see Cohen, 1992). This allowed us to convert the SMEs' responses to ds. In terms of the number of accidents associated with each descriptor, the ds were translated into the number of accidents using the z table.