Using neural networks to predict high-risk flight environments from accident and incident data

ABSTRACT

Flight risk assessment tools (FRATs) aid pilots in evaluating risk arising from the flight environment. Current FRATs are subjective, based on linear analyses and subject-matter expert interpretation of flight factor/risk relationships. However, a ‘flight system’ is complex with non-linear relationships between variables and emergent outcomes. A neural network was trained to categorize high and low-risk flight environments from factors such as the weather and pilot experience using data extracted from accident and incident reports. Negative outcomes were used as markers of risk level, with low severity outcomes representing low-risk environments and high severity outcomes representing high-risk environments. Eighteen models with varied architectures were created and evaluated for convergence, generalization and stability. Classification results of the highest performing model indicated that neural networks have the ability to learn and generalize to unseen accident and incident data, suggesting that they have the potential to offer an alternative to current risk analysis methods.

Keywords:

• neural networks
• flight safety
• flight risk
• modelling
• safety management

1. Introduction

In the aviation industry, risk management has become increasingly formalized and regulated. Many US Part 91 operators (covered by general operating and flight rules) and Part 135 operators (companies operating commuter and on-demand services) have voluntarily established risk assessment procedures [1]. In 2012, the International Civil Aviation Organization (ICAO) implemented Annex 19, requiring member states to mandate for the establishment of data-driven risk management processes for international operators of large and turbojet aircraft [2]. In 2014, the Federal Aviation Administration (FAA) implemented 14 Code of Federal Regulations (CFR) part 135.617 requiring helicopter air ambulance pilots to conduct formal pre-flight risk assessments [3]. In 2015, the FAA added part 5 to 14 CFR applicable to US part 121 (airline) and international General Aviation (GA) operators [4].

Current flight risk assessment tools (FRATs), such as that developed by the FAA [5], are typically based upon a subject-matter expert (SME) risk and hazard analysis of accident data and compute a ‘risk number’ from a linear combination of ‘risk values’ assigned to flight environment factors related to the aircraft, aircrew, resource and atmospheric states expected during the flight. However, aviation operations are complex, with emergent outcomes arising from inter-related variables with relationships that are likely to be non-linear, and neither completely deterministic nor completely random [6]. As a result, the use of linear methods may not be the most appropriate approach.

In a comparative study of maintenance operations, linear and non-linear analysis methods were applied to the prediction of safety outcomes collected over 6.2 years [7]. While Poisson regression (linear) completely failed to predict safety outcomes, neural networks (NNs) (non-linear) were significantly more accurate. This suggests that the analysis method should reflect the properties of the system being analysed and that linear methods ‘may be totally inappropriate if the underlying mechanism is nonlinear’ [8, p. 36]. Therefore, for the prediction of flight risk, an emergent property of a complex system, it is expected that a non-linear analysis method would be more successful.

Aviation risk-analysis research is dominated by methods dependent on SME knowledge, often incorporated into Bayesian models. However, expert knowledge of highly complex systems is limited since threat–consequence relationships can be ambiguous [9–11] and unknown or unexpected relationships cannot be accounted for [11]. Furthermore, while Bayesian methods may result in high-performing models if the prior probability is based on population statistics, in practice it is typically calculated from finite samples and/or SME knowledge. Therefore, the applicability of Bayesian networks is limited [12]. Fuzzy logic has also been applied to the analysis of flight environment factors to assess risk (e.g., [13,14]). However, in concluding his discussion of fuzzy logic methods, Hadjimichael [13, p.6516] stated that ‘a more robust method of determining the “most causal” risk factors is necessary. This is a complex issue, as finding a meaningful and useful definition of “most causal” is a significant research challenge’.

In contrast, NNs do not require prior assumptions or expert knowledge of a system. A NN performs a holistic analysis of historical data to determine an overall relationship between multiple inputs and multiple outputs. NNs are based upon a simplified model of human NNs, containing nodes, which represent input and output values (derived from a learning data set), and ‘hidden’ layers that allow complex, non-linear relationships to be modelled. The essential feature of NNs is that they learn the relationship(s) between variables in a building process, and self-correct. They are trained by exposure to historical data presented in a supervised leaning set, with known inputs and outputs. Once data from the supervised learning set have passed through the model, the error between actual and expected output is calculated and the weights in the model are revised iteratively through a back-propagation process until the solution converges and the overall error rate falls below a pre-defined criterion. To validate the NN, the derived model is then supplied with input data from an unseen (hold-out) data set and the output predictions from the network are compared to the known actual outputs [15,16].

In recent years, NNs have emerged as a data mining technique for exploring complex relationships in large data sets (e.g., [16,17]). NNs allow the simultaneous prediction of multiple outcomes from multiple inputs, and hence are ideal for predicting flight risk from flight environment factors. They are particularly well suited to applications with noisy, missing, overlapping, non-linear and non-continuous data [18] and can also handle highly unstructured data typical of that derived from accident and incident reports. They provide a means of building an empirically describable and verifiable model, which allows for the incorporation of contextual information [19].

NNs have been applied to a diverse range of classification problems such as the stock market [20] and cancer survival [21]. In the transportation field, they have been successfully applied to specific complex prediction problems. A decision-making tool for aircraft safety inspectors [22] evaluated the ability of a hybrid two-stage NN to analyse the relationships between aircraft operation and maintenance data, and service difficulty reporting (SDR) profiles. When compared with actual SDR numbers, 13 out of 19 NN models developed had R² values above 0.80. Classifications of marine accidents based on the river stage, traffic level, utilizations, location, weather and time were performed with 80% accuracy [23]. Predictions of the landing speed of McDonnell-Douglas MD80 aircraft in no/low-gust and high-gust conditions based on airport topology, the environment and flight and aircraft parameters were 95% correct [24]. Predictions of pilot decision-making in the resolution of a disruptive passenger incident [16] were 100% correct. Liu et al. [25] developed a NN to predict fatal accidents in GA operations from an analysis of flight environment-type factors and achieved a classification accuracy of over 78%. Harris and Li [26] developed a NN model based on the theoretical model of error causation underpinning the human factors analysis and classification system (HFACS) [27,28]. They found that 74% of unsafe acts (errors) implicated in 523 military aviation accidents could be correctly predicted from their preconditions.

The aim of this study was to investigate the ability of a NN to classify flight environments according to the level of risk. Since risk itself is ‘multidimensional and nuanced’ [29, p.1647], and therefore difficult to define, accidents and incidents – classified as having high and low severity outcomes – were used as markers of risk level. The NN was trained to predict outcome severity using data extracted from historic accident and incident reports. Model development and training was based on the four-stage process described by Diallo [24]: selecting and defining the input and output nodes; data source selection and coding; building the model; and model evaluation.

2. Method

2.1. Stage 1: selecting and defining the input and output nodes

An initial set of 37 input variables was derived from the FAA’s FRAT [5], the Aviation Safety Reporting System (ASRS) coding taxonomy [30], the ICAO (1993) human factors checklist [31] and the software–hardware–environment liveware (SHEL[L]) model [31,32]. After a review of the National Transportation Safety Board (NTSB) and ASRS reports, two further variables were included – ‘high altitude’ and ‘busy/complex airspace’ – both of which were frequently present at the time of the accident or incident. After initial coding, the number of variables was subsequently reduced as a result of missing data and infrequently occurring factors, resulting in 29 input variables (see Table 1).

Table 1. Definition of input variables.

Input	Operational definition
Single pilot	Operations in which a second pilot is not present Operations in which a second pilot is present, but not trained or qualified in accident/incident aircraft type
Inexperienced First Officer	First Officer with less than 200 h in accident/incident aircraft type
Captain less than 200 h in type	Captain with less than 200 h in accident/incident aircraft type
Captain less than 100 h in 90 days	Captain with less than 100 h in the 90 days prior to the accident/incident in any aircraft
Low light/night	Night, dawn, dusk or low light reported for entire or part of operation
Weather below visual flight rules	Ceilings 915 m or less and/or visibility 8 km or less at either departure or destination airport
Surface winds greater than 15.4 m·s^−1	Winds (including gusts) reported greater than 15.4 m·s^−1 at departure or destination at time of accident/incident
Crosswinds greater than 7.7 m·s^−1	Crosswinds (including gusts) reported greater than 7.7 m·s^−1 at departure or destination at time of accident/incident
Heavy rain	Heavy rain reported at or in the vicinity of departure or destination airport
Runway surface condition	Runway surface other than dry concrete
Uncontrolled airport	No operating control tower at departure or destination airport
No weather reporting	No weather reporting system located at departure or destination airport
Short runway	Runway shorter than 1524 m Runway reported as short for aircraft type
Airport elevation greater than 1524 m	Airport elevation greater than 1524 m MSL
Terrain	Airports defined as mountainous airports per ICAO Document 8168 Rising terrain reported as a factor in the accident/incident Operations involving arrival or departure procedures over terrain with elevation changes of 915 m or greater from airport elevation
High-altitude flight	Operations at altitudes higher than Flight Level 250 (circa. 7620 m)
Busy/complex airspace	Flight in or under Class B airspace Flight at airports with more than two runways Reported high-density traffic
Non-precision approach	Approach used does not have a glideslope
Circling approach	Approach used is not aligned with landing runway
Visual approach	No instrument approach is available at destination airport Instrument approach is available but pilots are cleared a visual approach in VMC conditions
Icing conditions	Reported flight conditions involve IMC flight and temperatures 10 °C and below Ground conditions require de-icing and/or anti-icing prior to take-off Use of aircraft anti-icing and de-icing systems is reported
Thunderstorms/wind shear	Reported thunderstorms and/or wind shear Thunderstorms and/or wind shear reported on the METAR at departure or destination
Turbulence	Reported turbulence
Reposition flight	Operations, other than personal flights, that do not involve the carriage of revenue passengers or cargo
International	Flights that either originate or terminate outside the contiguous USA, Hawaii or Alaska
Training	Flights involving the training of flight crew Initial operating experience Check rides and line checks
Maintenance factors	Deferred maintenance items on aircraft First flight of aircraft after maintenance has been performed Maintenance test or functional check flight
Time pressure	Flights involving a delay Medevac operations Time pressure reported by pilot
Personal factors	Reported sleep depravation Reported pilot personal stress or inter-crew stress Flight time greater than 5 h in duty period Duty period greater than 12 h

Note: All units are expressed as SI values. ICAO = International Civil Aviation Organization; IMC = instrument meteorological conditions: METAR = meteorological aerodrome report: MSL = mean sea level: VMC = visual meteorological conditions.

Commonly used FRATs typically categorize flight risk at two levels: high risk that requires threat mitigation and management consultation; and low risk that does not. Therefore, the outputs of the NN were chosen to similarly categorize flight outcomes at two levels. High severity outcomes (moderate, major and catastrophic events) were considered markers of high-risk environments; low severity outcomes (negligible and minor events) were attributed to low-risk environments. These outputs were defined by a combination of the levels of severity definitions found in the ICAO [2] safety risk severity table and example severity table (see Table 2).

Table 2. Output variables and definitions taken from the example severity table (ICAO, p.2,App 2–3) and the safety risk severity table (ICAO, p.2–29) [2].

Output	Initial categorization	ICAO definitions
Low severity consequence	Insignificant (0.2)	No significance to aircraft-related operational safety	Negligible Few consequences
	Minor (0.4)	Degrades or affects normal aircraft operational procedures or performance	Nuisance Operating limitations Use of emergency procedures Minor incident
High severity consequence	Moderate (0.6)	Partial loss of significant/major aircraft systems or results in abnormal application of flight operations procedures	Major Significant reduction in safety margins, a reduction in the ability of the operators to cope with adverse operating conditions as a result of an increase in workload or as a result of conditions impairing their efficiency Serious incident Injury to persons
	Major (0.8)	Complete failure of significant/major aircraft systems or results in emergency application of flight operations procedures	Hazardous A large reduction in safety margins, physical distress or a workload such that the operators cannot be relied upon to perform their tasks accurately or completely Serious injury Major equipment damage
	Catastrophic (1.0)	Loss of aircraft or lives	Catastrophic Equipment destroyed Multiple deaths

Note: Outcome severity is included in parentheses. ICAO = International Civil Aviation Organization.

2.2. Stage 2: data source selection and coding

The performance of a NN depends on the input and output combinations contained in the training data [33]. Therefore, to predict flight-outcome severity, the data needed to encompass a range of events from insignificant incidents to catastrophic accidents. NTSB investigations involve accidents having substantial consequences for participants and equipment (aircraft and property damage, casualties and fatalities). Events reported to the ASRS programme typically involve minor incidents (e.g., altitude and airspace deviations) that, although undesirable, generally result in little or no consequence. Therefore, the appropriate range of data were obtained from a combination of NTSB and ASRS reports extracted from their online databases.

This study focused on US 14 CFR Part 135 (scheduled and non-scheduled) and Part 91 (general) operations involving two-engine turbofan, turbojet and turboprop airplanes certified under US 14 CFR Part 25 (light and medium transport only) and Part 23 (normal category airplanes) [4]. Narrowing the focus to similar types of operation had the advantage of making certain factors approximately constant (e.g., level of training), thus reducing the number of input variables. Report selection was limited to these categories. Reports were rejected if they involved the following:

• illegal activity, e.g., use of illicit drugs;

• system failures whose cause or handling was not affected by factors encountered between engine start and engine shutdown;

• military, government, skydiving and low-altitude operations;

• experimental aircraft;

• a captain who did not hold a commercial pilot or air transport pilot certificate.

The data extraction and coding process was undertaken in two steps. In step one, all initial 39 input factors were coded. Categorical factors, e.g., single pilot, were recorded using a binary code, with ‘1’ indicating its presence and ‘0’ indicating its absence. For scale data, e.g., runway length, the value was recorded. Outcome severity was coded in increments of 0.2, with 0.2 indicating an insignificant outcome and 1.0 indicating a catastrophic outcome (see Table 2). Depending on the nature of the event, the most appropriate of column one or column two of Table 2 was used to determine outcome severity. Where necessary, the NTSB and ASRS reports were supplemented with airport, terrain, navigation and weather data from SkyVector [34], AirNav [35] and Ogimet [36].

While NTSB reports include information describing the entire flight, ASRS reports are de-identified and typically describe only the circumstances surrounding the incident itself. Therefore, they frequently lack data such as departure and destination airport information and pilot flight time. Additionally, both NTSB and ASRS reports yielded low numbers of personal aircrew factors, such as fatigue and stress. The following steps were performed to address these issues:

• Departure and destination airport variables selected in stage 1 were combined and re-defined as a single ‘airport’ variable (e.g., uncontrolled airport) that was applied to either airport, provided the missing airport data were not directly associated with the accident or incident. With the exception of reports concerning en-route events, if both the departure and destination airport information was missing, the report was rejected.

• For reports concerning en-route incidents containing neither departure nor destination information and for which the airports were not directly associated with the incident, all airport factors were coded as ‘absent’. This allowed the inclusion of high altitude and en-route incidents.

• ‘First Officer (FO) less than 200 h in type’ and ‘FO less than 100 h in 90 days’ were combined into a single variable of ‘Inexperienced FO’. If FO hours were not reported, this factor was coded as absent unless the narrative indicated that the FO was inexperienced.

• Flight time accrued during the duty period, sleep deprivation, personal stress and intra-crew stress were combined into a single ‘Personal factors’ input.

Scalar values were then re-coded as binary categorical factors according to the discriminator indicated by its definition. The output factors were also re-coded as binary categorical factors, with 0.5 discriminating between low and high severity outcomes.

A total of 467 reports meeting the operations and airplane criteria were retrieved from the NTSB database, involving events that occurred between January 2005 and December 2015. Of these, 206 reports were rejected. A further 1614 reports meeting the operations criteria were retrieved from the ASRS database, involving events that occurred between January 2010 and December 2015. Of these, 1220 reports were rejected. As a result, a total of 655 reports were coded and analysed.

2.3. Stage 3: building the model

The NN analysis tools in SPSS version 24.0 were used to build and evaluate multiple models.

Severity of outcome was entered as the dependent variable. The data set was split into three partitions: ‘training’, ‘testing’ and ‘hold-out’. Accident/incident data sets were randomly assigned to each partition, but the same ratio of NTSB to ASRS reports was maintained in each partition (see Table 3).

Table 3. Assignment of data sets.

		Number of reports
Partition	Proportion of data (%)	NTSB	ASRS
Training	60.2	157	203
Testing	29.8	78	102
Hold-out	10	26	34

Note: ASRS = Aviation Safety Reporting System; NTSB = National Transportation Safety Board.

Selecting the initial parameters for a NN architecture (such as the number of hidden layers and/or the number of nodes in each hidden layer) has been described as both ‘an empirical procedure with guidelines’ [7, p.5] and ‘more of an art than a science’ [8, p.42]. While rules of thumb guide NN design, the optimum combination of layers, nodes, activation functions and training settings for a particular problem is generally determined through trial and error. Therefore, multiple models were developed using either the automatic architecture option or the custom selections. Models designed using the custom options varied in choice of activation function of the output layer (identity, softmax, hyperbolic tangent or sigmoid) and the number of nodes in the hidden layer (5, 14 or 29). The hyperbolic tangent function was selected as the activation function for the hidden nodes for all models.

The gradient-descent training algorithm was used for all models. The maximum number of steps without an error was set at three. The minimum relative change in training error was set to 0.0001 and the training error ratio was 0.001.

Each NN model analysed the data 10 times, five times with the inputs ordered as presented in Table 1 and five times with the inputs reversed. The mean, median and standard deviation for the percentage of correct predictions were calculated from the 10 runs of each model. Mean importance values for each variable were calculated from the values reported for each run.

The best performing model was selected based on a combination of highest mean accuracies for the hold-out sample, lowest standard deviations and greatest area under the receiver operator characteristic (ROC) curve.

3. Results

Twenty NN models were created. When tested on the hold-out partition, 15 models had average classification accuracies of over 60% for both low and high severity outcomes. Three models had accuracies of over 65%. The highest performing model predicted the outcomes of unseen data with an average accuracy of 69.8% (SD 2.8%) for high severity outcomes and 65.5% (SD 8.4%) for low severity outcomes, with the highest performing run having a prediction accuracy of 71.1% for high severity outcomes and 77.3% for low severity outcomes. The mean area under the ROC curves was 0.843 for both low and high severity outcomes.

The average classification results of this model and the results of the highest performing run are presented in Tables 4 and 5. The architecture of the highest performing model had 58 nodes in the input layer, 29 nodes in the hidden layer and two nodes in the output layer. Mean variable rankings, importance values and normalized mean importance values are presented in Table 6.

Table 4. Average classification accuracy for the highest performing model.

		Classification accuracy
Data sample	Flight outcome	Mean (%)	Standard deviation (%)
Training	Low severity	75.3	3.6
	High severity	77.4	1.6
Testing	Low severity	75.1	4.7
	High severity	75.1	4.1
Hold-out	Low severity	65.5	8.4
	High severity	69.8	2.8

Table 5. Classification accuracy of the highest performing run of the highest performing model.

			Predicted
Sample			Low severity	High severity	Correct (%)
Training	Observed	Low severity	131	39	77.1
		High severity	45	146	76.4
		Overall			76.7
Testing	Observed	Low severity	55	17	76.4
		High severity	28	79	73.8
		Overall			74.9
Hold-out	Observed	Low severity	17	5	77.3
		High severity	11	27	71.1
		Overall			73.3

Table 6. Variable importance values.

Variable	Mean importance	Standard deviation	Normalized mean importance
Busy/complex airspace	0.059	0.012	100.0
Uncontrolled airport	0.055	0.012	93.9
Visual approach	0.054	0.008	91.2
Runway surface condition	0.049	0.013	82.4
Maintenance factors	0.049	0.017	82.2
Single pilot	0.046	0.010	77.3
Surface winds greater than 15.4 m·s^−1	0.045	0.017	75.4
Circling approach	0.040	0.016	66.9
Thunderstorms/wind shear	0.039	0.014	65.4
Weather below visual flight rules minima	0.036	0.012	60.7
Training	0.035	0.013	59.7
Personal factors	0.035	0.017	59.0
Crosswinds greater than 7.7 m·s^−1	0.035	0.007	58.8
Non-precision approach	0.034	0.010	57.8
Inexperienced First Officer	0.033	0.007	55.6
Short runway	0.031	0.013	53.1
Icing conditions	0.030	0.009	50.8
Heavy rain	0.030	0.005	50.5
Airport elevation greater than 1524 m	0.030	0.011	50.0
Time pressure	0.029	0.009	49.2
High-altitude flight	0.026	0.005	44.7
International	0.026	0.005	43.2
Reposition flight	0.025	0.008	41.7
Captain less than 200 h in type	0.024	0.007	40.8
Turbulence	0.024	0.007	39.8
No weather reporting	0.023	0.007	39.5
Terrain	0.023	0.007	38.6
Low light/night	0.020	0.003	34.4
Captain less than 100 h in 90 days	0.019	0.002	31.5

Note: All units are expressed as SI values.

4. Discussion

Adya and Colopy [37] suggest that a ‘good’ prediction model should demonstrate good training sample performance, be able to accurately predict outcomes from unseen data (good generalization) and be stable (produce consistent results). When measured against these standards, the results indicate that NNs have the potential for becoming ‘good’ models for categorizing flight environments. The training and test results of the highest performing model show that the model was able to learn, and the hold-out accuracy of the highest performing run suggests that the model has the potential to generalize well to unseen data (see Table 6). The areas under the ROC curves indicate that the NN has a better than 0.8 probability of correctly categorizing a random event and is, therefore, a ‘good’ discriminator of low and high-risk flight environments. However, the standard deviations and hold-out results suggest that the model may be sensitive to the data set and variable input order, particularly for low severity outcomes but less so for more critical high severity outcomes.

Colopy et al. [38] suggest that a NN should be validated by a comparison of out-of-sample performance with other well-accepted models. However, in this case, selecting such models against which to make an informative comparison proved problematic. Typical pre-flight risk analyses (e.g., those similar to the FAA’s FRAT) are generally tailored to the specific needs of an organization. They are highly subjective, depending on the choice of inputs, the scores assigned and the operators’ tolerance of risk. Additionally, there is no universally accepted measure of high and low risk.

However, a comparison with the GA fatalities NN, although limited by fundamental differences such as type of aviation operation and inputs analysed, was possible to some degree. While the average training and test results for the highest performing model were similar to the accuracy achieved by Liu et al. [25], the average hold-out accuracy was a little less favourable. This may be explained, however, by the broader range of causal factors analysed by the GA network, which, while perhaps being good indicators of risk (e.g., phase of flight), would not be suitable for pre-flight risk analysis.

With no well-accepted risk assessment model against which to validate the NN, the results were benchmarked against the prediction accuracies of two other complex systems, the weather and the stock market. A NN developed to predict the height of the 500-hPa pressure level in the northern hemisphere achieved an average monthly prediction accuracy of between 0.70 and 0.95 [39]. In contrast, the stock market, which is highly reactive to human emotions and decisions, has an average prediction accuracy of between 50.8 and 65.79% [20]. These studies indicate that, first, the ability to predict the outcomes of any complex system is limited and, second, the more it involves human agents, the less predictable it gets. However, research also indicates that this effect may be moderated by agent training and the resolution of the analysis. A NN applied to a single aviation decision made by highly trained airline pilots predicted outcomes with 100% accuracy [16]. In contrast, the GA fatalities NN [25] applied to a more macroscopic element of the aviation system, involving pilots who were likely, in general, less well trained, achieved only 78.9% accuracy. These results imply that, for systems involving human agents, higher resolution (more specific problems), better training and less freedom of choice increases predictability.

The flight environment involves a mix of the human, the machine, resources and the atmosphere. The predictability of these components runs on a continuum from highly predicable (the machine), to somewhat predictable (the atmosphere and resources) to less predictable (the human). However, the humans in the professional aviation system (pilots, air traffic controllers, maintainers, etc.) are well trained and their range of choices is restrained by regulation. Against this context, the results obtained in this study, better than stock market prediction but inferior to the prediction of weather events, are considered reasonable for an initial study of a NN flight system analysis. However, it is acknowledged that, before a NN can be applied to real-world pre-flight risk assessment tasks, where high-risk ‘misses’ could have significant negative consequences, further refinements are required.

5. Recommendations and conclusion

Given the ambiguous nature of risk and the complexity of the flight system, the development and validation of accurate and meaningful pre-flight risk analysis tools is challenging. However, the findings of this exploratory study indicate that NNs are both suited to and have the potential for such a task. The classification accuracies, although limited by system complexity, are commensurate with those of other complex systems and similar research.

In addition to categorizing flight environments, NNs potentially offer a means of identifying the most significant predictors of flight risk through a sensitivity analysis of the variables (see Table 6). The trends, similarities and differences in the variable rankings found in this study and between other studies may provide insight into the underlying features of the aviation system.

It is recommended that future research be directed down three paths. The first aims to improve the convergence and stability of the NN by addressing the variable, missing data, data source and coding problems. The current research can only be considered as a proof of concept – the current data set is quite small. As a result, the second recommendation is to expand the data set to include accident and incident reports from other countries, which can be used in the process of training, validation and testing of NN. Building on this, the third recommendation focuses on improving accuracy and generalization through NN design, allowing a more inclusive analysis of the flight system encompassing supervision and organizational elements. Suggestions for such methods include a ‘zooming out’ approach involving the analysis of individual system elements through multiple and multi-stage NNs and the output of risk indicators that feed forward into a flight-risk analysis (combining the methodologies applied by Luxhøj and Williams [22] and Hsiao et al. [7]). It is expected that progress made through such research could lead towards the development of functional, objective and informative flight-risk analysis tools to aid pilot decision-making.

Disclosure statement

No potential conflict of interest was reported by the authors.