A Survey Methodology for Measuring Safety-Related Trends in the National Airspace System

Abstract

Making informed judgments about the effects of technological, human, or environmental changes on civil aviation requires reliable information. The National Aviation Operational Monitoring Service (NAOMS), a research project under NASA’s Aviation Safety Program, developed and tested a survey methodology designed to provide statistically reliable information on changes over time in safety-related events in the national airspace. Such information would aid decision makers in determining what areas required attention. To evaluate the NAOMS concept, data from nearly 20,000 randomly selected air-carrier pilots were collected over 3 years. Results demonstrate that the NAOMS approach can reliably identify changes over time in the rates of safety-related events.

Notes

¹ Internet surveys were not in wide use when the NAOMS project was being planned. Following the completion of data collection for the main air carrier survey, telephone and Internet modes were compared. One thousand pilots were randomly selected from the NAOMS survey pool and invited to participate in the Internet survey. Over a 2-month period, only 128 pilots completed the Internet survey, for a response rate of 13% compared to a response rate of 81% for the pilots contacted for the telephone survey.

² According to federal regulations, only a Commercial Pilot certificate and a Second Class medical certificate are required to fly for hire. However, most operators require or strongly encourage job applicants to have an ATP and a First Class medical. Furthermore, pilots are required to have these certificates to serve as pilot-in-command of many operations. Some commercial operations use single-engine aircraft. However, most of the aircraft used for commercial operations have multiple engines and most pilots intent on pursuing careers as commercial pilots obtain multiengine ratings during training. Few modern aircraft require a flight engineer. However, to obtain the Flight Engineer certificate, a pilot must demonstrate knowledge of turbine systems. Hence, many operators require or strongly encourage job applicants to have Flight Engineer certificates. Because pilot training is typically less expensive in the United States than in other countries, many foreign pilots obtain U.S. pilot certificates. Limiting the sample to pilots based in the United States eliminated foreign pilots who had obtained U.S. pilot certificates but did not fly in the United States.

³ At the time of the NAOMS survey, few individuals relied solely on cell phones for telephone communication. If a NAOMS-like survey were to be conducted now, one would need to ensure that individuals who relied exclusively on cell phones for telephone service were included.

⁴ The negative binomial describes a family of distributions. The Poisson could be regarded as a special case. In Poisson and negative binomial regression, the natural logarithm of the count of the number of events is modeled as a linear function of the predictors. Measures of exposure (e.g., the number of legs flown) are entered as unparameterized predictors or “offset variables” in the model. Hence, the effect of the exposure is taken into account in estimating the effects of the other predictors.

⁵ Mean total hours: Match = 10,350, SD = 6,822; No match = 7,069, SD = 6,272; t(6,344) = 9.38, p < .001. Mean legs: Match = 38, SD = 35; No match = 69, SD = 59; t(419.62) = 10.25, p < .001. Mean recent hours: Match = 98, SD = 48; No match = 112, SD = 52; t(446.90) = 5.42, p < .001. Mean hours/leg: Match = 3.75, SD = 2.75; No match = 2.50, SD = 2.14; t(495.54) = 11.11, p < .001.

⁶ Aircraft size categories, χ²(3) = 798.4, p < .001; type of operation, χ²(1) = 58.2, p < .001; and positions flown, χ²(2) = 27.55, p < .001.

⁷ The rate of reported bird strikes increased across the observation period, b = .020, Wald χ²(1) = 8.227, p = .004. However, this trend is dependent on an usually low rate of reported bird strikes during the first quarter of 2002 and an unusually high rate of reported bird strikes during the final quarter of 2004. If these quarters are removed from the data, the observed increase is not significant, Wald χ²(1) = .005, p = .945. Also, there were significant differences in the rates of reported bird strikes between aircraft categories, Wald χ²(3) = 140.781, p < .001. The highest rates of bird strikes were reported by pilots of wide-body aircraft. Pilots of large aircraft reported significantly lower rates of bird strikes than did pilots of wide-body aircraft, Wald χ²(1) = 43.724, p < .001. Pilots of medium aircraft reported lower rates than did pilots of large aircraft, but this difference was only marginally significant, Wald χ²(1) = 3.197, p = .074. The rates of reported bird strikes did not differ between medium and small aircraft categories, Wald χ²(1) = .011, p = .915. In addition, pilots of aircraft engaged in cargo operations reported significantly higher rates of bird strikes than did pilots of passenger aircraft, Wald χ²(1) = 384.387, p < .001.

⁸ There were significant differences in the rate of reported events across aircraft categories, Wald χ²(3) = 141.374, p < .001. Pilots of small aircraft reported significantly higher rates of events than did pilots of other aircraft, Wald χ²(1) = 58.625, p < .001. No other differences between aircraft categories were observed. Pilots of passenger aircraft reported slightly higher rates of these events than did pilots of cargo aircraft, but this difference was not statistically significant at conventional levels, Wald χ²(1) = 3.048, p = .081.