Situation Awareness and campaign assessments

Abstract

Military-led nation building campaigns can face many challenges as advances in certain aspects such as the economy and infrastructure may well be accompanied by setbacks in other areas such as security, making it difficult to judge overall progress. In this context, campaign assessments provide vital decision support for adapting strategy and deploying resources to best effect. The value of such assessments is enhanced when they report not only what has happened but why and when they also indicate likely future events—mirroring the three levels in Endsley’s model of Situation Awareness. Our aim is to develop a process for evaluating changes to the complex organisational structure and processes used for producing assessment documents, even if it is difficult to understand the linkages between such changes and output quality. Using 2 years’ worth of Australian Defence Force quarterly campaign assessments, we study the effect of adoption of a new analysis framework in their production. Using the conventions of thematic analysis and Endsley’s model of Situation Awareness, we introduce the Situation Awareness Elicitation method for obtaining metrics pertaining to the value of assessments before and after introduction of the framework and perform statistical analysis on this data.

Keywords:

• Situation Awareness
• thematic analysis
• organisational change

Acknowledgements

The authors would like to thank Jamie Watson and Andrew Haddock for commissioning this work, Alexander Kalloniatis for his guidance regarding analysis techniques, Irena Ali for her assistance in originally pointing us towards thematic analysis for data collection, Jason Cromarty, Alison Hickman and Scott Wheeler for fruitful discussions, and finally Peter Dortmans who provided constructive feedback on the work.

Notes

1 In our case study we distinguish between two roles played by a decision maker: that of a manager who strives to increase the value of campaign assessments through changes to organisational processes or structure (our primary interest), and that of a military commander who uses information in campaign assessments to respond appropriately to changes in the operational environment.

2 These instances we refer to as ‘hits’ in the following.

3 To prevent bias being introduced to coding, the coders were not involved in production of the assessments being studied.

4 A transparent, bottom-up method of assessing progress in a counter-insurgency campaign that captures embedded contextual data—both qualitative and quantitative—at each layer of assessment, from the tactical to the strategic (Connable, 2012).

5 See http://www.physics.csbsju.edu/stats/KS-test.html