Empirical evaluation of the Process Overview Measure for assessing situation awareness in process plants

Abstract

The Process Overview Measure is a query-based measure developed to assess operator situation awareness (SA) from monitoring process plants. A companion paper describes how the measure has been developed according to process plant properties and operator cognitive work. The Process Overview Measure demonstrated practicality, sensitivity, validity and reliability in two full-scope simulator experiments investigating dramatically different operational concepts. Practicality was assessed based on qualitative feedback of participants and researchers. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA in full-scope simulator settings based on data collected on process experts. Thus, full-scope simulator studies can employ the Process Overview Measure to reveal the impact of new control room technology and operational concepts on monitoring process plants.

Practitioner Summary: The Process Overview Measure is a query-based measure that demonstrated practicality, sensitivity, validity and reliability for assessing operator situation awareness (SA) from monitoring process plants in representative settings.

Keywords:

• Full-scope simulator
• experiments
• process control
• nuclear

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. At the BWR plant where operators were recruited, each crew consisted of a shift supervisor as well as reactor and TOs.

2. In this experiment, automation operated the reactor side of Plant B to start up the plant with the AO.

3. Note that the aggregation of Process Overview excludes data collected from the role of Shift-supervisor and WM in the Traditional and Untraditional staffing solution, respectively, because ASURS was intended to measure the Scenario Understanding of the operators only. Further, ASURS data from the first four crews (Plant 1) were omitted from the analysis because the process experts (i.e. raters) were retrained on the measurement protocol.

4. The Mann-Whitney U-test yielded a p-value of .09, which confirmed the main effect of staffing solution. However, both the parametric and non-parametric tests yielded p-values between .05 and .1, which were weak indications of experimental effect.

5. Note that correlations between measures should generally not be very strong (i.e. r² > .5) as that is often an indication that the measures are only superficially different, measuring the same underlying construct (i.e. lack of discriminant properties).