Tying up loose ends: a method for constructing and evaluating decision aids that meet blunt and sharp-end goals

Abstract

We present a methodological framework for constructing and evaluating decision aids – fast and frugal trees (FFTs) – ideally suited to the front line of an organisation. Their performance can be analysed in signal detection theory, allowing for transparent selection of FFTs given managerial-level trade-offs among type I and II errors. We extend FFTs from binary classification to selection from multiple actions (FFT multiple) as well as performance analysis to organisational goal states beyond type I and II error reduction. Concepts and framework are introduced and a tutorial-style example application (threat assessment at military checkpoints) is provided. Throughout, we discuss ways to deal with missing or incomplete data and show that the performance of decision aids may be overestimated if the effectiveness of actions is not heeded. The methodology can be used to construct and evaluate decision aids in any area characterised by dichotomised cues and a one-to-many mapping between categorisation outcomes and actions.

Abstract

Practitioner Summary: The paper presents a methodological framework for the construction of decision aids and their evaluation along multiple goal states across institutional levels. We then apply this framework to construct and evaluate decision aids for threat assessment in military operations. Ways to deal with missing and incomplete data are discussed.

Keywords::

• fast and frugal trees
• signal detection theory
• decision support
• military personnel