The use of control charts by laypeople and hospital decision-makers for guiding decision making

ABSTRACT

Graphs presenting healthcare data are increasingly available to support laypeople and hospital staff’s decision making. When making these decisions, hospital staff should consider the role of chance—that is, random variation. Given random variation, decision-makers must distinguish signals (sometimes called special-cause data) from noise (common-cause data). Unfortunately, many graphs do not facilitate the statistical reasoning necessary to make such distinctions. Control charts are a less commonly used type of graph that support statistical thinking by including reference lines that separate data more likely to be signals from those more likely to be noise. The current work demonstrates for whom (laypeople and hospital staff) and when (treatment and investigative decisions) control charts strengthen data-driven decision making. We present two experiments that compare people’s use of control and non-control charts to make decisions between hospitals (funnel charts vs. league tables) and to monitor changes across time (run charts with control lines vs. run charts without control lines). As expected, participants more accurately identified the outlying data using a control chart than using a non-control chart, but their ability to then apply that information to more complicated questions (e.g., where should I go for treatment?, and should I investigate?) was limited. The discussion highlights some common concerns about using control charts in hospital settings.

KEYWORDS:

• Statistics
• Decision making
• Healthcare

Acknowledgement

The views expressed in this publication are those of the author(s) and not necessarily those of the National Health Service (NHS), the National Institute for Health Research (NIHR), or the Department of Health. We also thank Richard Lilford for his encouragement throughout this project.

Notes

¹One may note that hospital data are often presented as tables and that people often prefer tables over graphs (Hildon et al., 2012). Past research comparing people's use of tables and graphs has found that the two presentation methods are best suited to answer different types of questions (Speier, 2006; Vessey, 1991). While tables help decision-makers better identify past, unique data, graphs are better at portraying patterns in the data—for example, “In what month was the percent of patients waiting more than 4 hours to be seen in A&E highest?” versus “Is this percent increasing?”. As quality improvement relies on recognizing patterns in data, we concentrate on graphs in the current work.

²For simplicity, the current study focuses on a single type of control chart and a single Western electronic rule to identify irregular data—that is, any data point outside 3 standard deviations is irregular. More complicated run charts that include multiple sets of control lines (e.g., one set at 2 SDs and another set at 3 SDs) enhance decision-makers' ability to identify lower level statistically irregular trends. For example, while one data point outside of 3 standard deviations is statistically irregular, two consecutive data outside 2 standard deviations are irregular. While it is tempting to apply as many rules as possible, decision-makers ought to remain cautious as the more rules applied, the greater the probability of a false positive. For more information, see Amin, 2001.

³As we wanted the percentages displayed in the charts to convey something that could actually happen, whole numbers were needed for the numerator of each hospital's performance. To do this we first found the flipped percentages. From the original data set, we deducted each hospital's percentage compliance from its grand mean. Then we deducted these differences from the grand mean. Then, second, to find the new numerator, we multiplied each hospital's denominator by its flipped percentage and rounded to the nearest whole number. The second funnel chart contains each hospital's percentage compliance based on the rounded numerator and the original denominator.

⁴As we wanted the percentages displayed in the charts to convey something that could actually happen, we followed the same procedure as that described in Footnote 3 to flip the run chart data.

Additional information

Funding

This work was supported by National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care West Midlands initiative [wmclahrc-2014-1].