Using neural networks to predict high-risk flight environments from accident and incident data

Figures & data

Table 1. Definition of input variables.

Input	Operational definition
Single pilot	Operations in which a second pilot is not present Operations in which a second pilot is present, but not trained or qualified in accident/incident aircraft type
Inexperienced First Officer	First Officer with less than 200 h in accident/incident aircraft type
Captain less than 200 h in type	Captain with less than 200 h in accident/incident aircraft type
Captain less than 100 h in 90 days	Captain with less than 100 h in the 90 days prior to the accident/incident in any aircraft
Low light/night	Night, dawn, dusk or low light reported for entire or part of operation
Weather below visual flight rules	Ceilings 915 m or less and/or visibility 8 km or less at either departure or destination airport
Surface winds greater than 15.4 m·s^−1	Winds (including gusts) reported greater than 15.4 m·s^−1 at departure or destination at time of accident/incident
Crosswinds greater than 7.7 m·s^−1	Crosswinds (including gusts) reported greater than 7.7 m·s^−1 at departure or destination at time of accident/incident
Heavy rain	Heavy rain reported at or in the vicinity of departure or destination airport
Runway surface condition	Runway surface other than dry concrete
Uncontrolled airport	No operating control tower at departure or destination airport
No weather reporting	No weather reporting system located at departure or destination airport
Short runway	Runway shorter than 1524 m Runway reported as short for aircraft type
Airport elevation greater than 1524 m	Airport elevation greater than 1524 m MSL
Terrain	Airports defined as mountainous airports per ICAO Document 8168 Rising terrain reported as a factor in the accident/incident Operations involving arrival or departure procedures over terrain with elevation changes of 915 m or greater from airport elevation
High-altitude flight	Operations at altitudes higher than Flight Level 250 (circa. 7620 m)
Busy/complex airspace	Flight in or under Class B airspace Flight at airports with more than two runways Reported high-density traffic
Non-precision approach	Approach used does not have a glideslope
Circling approach	Approach used is not aligned with landing runway
Visual approach	No instrument approach is available at destination airport Instrument approach is available but pilots are cleared a visual approach in VMC conditions
Icing conditions	Reported flight conditions involve IMC flight and temperatures 10 °C and below Ground conditions require de-icing and/or anti-icing prior to take-off Use of aircraft anti-icing and de-icing systems is reported
Thunderstorms/wind shear	Reported thunderstorms and/or wind shear Thunderstorms and/or wind shear reported on the METAR at departure or destination
Turbulence	Reported turbulence
Reposition flight	Operations, other than personal flights, that do not involve the carriage of revenue passengers or cargo
International	Flights that either originate or terminate outside the contiguous USA, Hawaii or Alaska
Training	Flights involving the training of flight crew Initial operating experience Check rides and line checks
Maintenance factors	Deferred maintenance items on aircraft First flight of aircraft after maintenance has been performed Maintenance test or functional check flight
Time pressure	Flights involving a delay Medevac operations Time pressure reported by pilot
Personal factors	Reported sleep depravation Reported pilot personal stress or inter-crew stress Flight time greater than 5 h in duty period Duty period greater than 12 h

Note: All units are expressed as SI values. ICAO = International Civil Aviation Organization; IMC = instrument meteorological conditions: METAR = meteorological aerodrome report: MSL = mean sea level: VMC = visual meteorological conditions.

Table 2. Output variables and definitions taken from the example severity table (ICAO, p.2,App 2–3) and the safety risk severity table (ICAO, p.2–29) [2].

Output	Initial categorization	ICAO definitions
Low severity consequence	Insignificant (0.2)	No significance to aircraft-related operational safety	Negligible Few consequences
	Minor (0.4)	Degrades or affects normal aircraft operational procedures or performance	Nuisance Operating limitations Use of emergency procedures Minor incident
High severity consequence	Moderate (0.6)	Partial loss of significant/major aircraft systems or results in abnormal application of flight operations procedures	Major Significant reduction in safety margins, a reduction in the ability of the operators to cope with adverse operating conditions as a result of an increase in workload or as a result of conditions impairing their efficiency Serious incident Injury to persons
	Major (0.8)	Complete failure of significant/major aircraft systems or results in emergency application of flight operations procedures	Hazardous A large reduction in safety margins, physical distress or a workload such that the operators cannot be relied upon to perform their tasks accurately or completely Serious injury Major equipment damage
	Catastrophic (1.0)	Loss of aircraft or lives	Catastrophic Equipment destroyed Multiple deaths

Note: Outcome severity is included in parentheses. ICAO = International Civil Aviation Organization.

Table 3. Assignment of data sets.

		Number of reports
Partition	Proportion of data (%)	NTSB	ASRS
Training	60.2	157	203
Testing	29.8	78	102
Hold-out	10	26	34

Note: ASRS = Aviation Safety Reporting System; NTSB = National Transportation Safety Board.

Table 4. Average classification accuracy for the highest performing model.

		Classification accuracy
Data sample	Flight outcome	Mean (%)	Standard deviation (%)
Training	Low severity	75.3	3.6
	High severity	77.4	1.6
Testing	Low severity	75.1	4.7
	High severity	75.1	4.1
Hold-out	Low severity	65.5	8.4
	High severity	69.8	2.8

Table 5. Classification accuracy of the highest performing run of the highest performing model.

			Predicted
Sample			Low severity	High severity	Correct (%)
Training	Observed	Low severity	131	39	77.1
		High severity	45	146	76.4
		Overall			76.7
Testing	Observed	Low severity	55	17	76.4
		High severity	28	79	73.8
		Overall			74.9
Hold-out	Observed	Low severity	17	5	77.3
		High severity	11	27	71.1
		Overall			73.3

Table 6. Variable importance values.

Variable	Mean importance	Standard deviation	Normalized mean importance
Busy/complex airspace	0.059	0.012	100.0
Uncontrolled airport	0.055	0.012	93.9
Visual approach	0.054	0.008	91.2
Runway surface condition	0.049	0.013	82.4
Maintenance factors	0.049	0.017	82.2
Single pilot	0.046	0.010	77.3
Surface winds greater than 15.4 m·s^−1	0.045	0.017	75.4
Circling approach	0.040	0.016	66.9
Thunderstorms/wind shear	0.039	0.014	65.4
Weather below visual flight rules minima	0.036	0.012	60.7
Training	0.035	0.013	59.7
Personal factors	0.035	0.017	59.0
Crosswinds greater than 7.7 m·s^−1	0.035	0.007	58.8
Non-precision approach	0.034	0.010	57.8
Inexperienced First Officer	0.033	0.007	55.6
Short runway	0.031	0.013	53.1
Icing conditions	0.030	0.009	50.8
Heavy rain	0.030	0.005	50.5
Airport elevation greater than 1524 m	0.030	0.011	50.0
Time pressure	0.029	0.009	49.2
High-altitude flight	0.026	0.005	44.7
International	0.026	0.005	43.2
Reposition flight	0.025	0.008	41.7
Captain less than 200 h in type	0.024	0.007	40.8
Turbulence	0.024	0.007	39.8
No weather reporting	0.023	0.007	39.5
Terrain	0.023	0.007	38.6
Low light/night	0.020	0.003	34.4
Captain less than 100 h in 90 days	0.019	0.002	31.5

Note: All units are expressed as SI values.

International Civil Aviation Organization. Safety Management Manual Doc 9859. 3rd ed. Montreal (CA): International Civil Aviation Organization ; 2012 [cited 2021 Jan 2]. Available from: https://www.icao.int/sam/documents/rst-smsssp-13/smm_3rd_ed_advance.pdf

 Google Scholar