Better Quality Metrics Could Illuminate Quality-Efficiency Tradeoffs in Operating Room Management

This article is referred to by:

Perioperative Efficiency vs. Quality of Care – Do We Always Have to Choose?

Chernov et al. [1] describe their experience with the institution of a system of operating room (OR) management designed to improve efficiency and report that they did not experience an adverse impact on quality. Chernov defined quality based on the frequency of a series of 95 “harmful events or deviations from industry-accepted standards of care” and relied on event reporting to monitor their frequency. In this report, they specifically focused on 10 such uncommon “events”, many of which reflect process deviations rather than outcomes and they do not present frequencies of the other 85 “events”. One could thus quibble about their choices of quality measures. In addition, identification of these events seems to have depended on self-reporting by members of the OR team, who might have been more reluctant to report in the face of increased managerial scrutiny. Indeed, efficiency can itself be measured in many different ways [2] and others might disagree with Chernov et al.’s [1] efficiency metrics as well. Nevertheless, Chernov’s data does not suggest any increase in these events after implementation of the managerial system despite decreased overtime and turnover time and improved first case start times.

These results should be somewhat reassuring to colleagues at Chernov’s institution and to the rest of us. However, they may prompt a more philosophical view of these issues. Efficiency represents the maximal ratio of outputs to inputs, as compared to productivity which is simply the measurement of output [3]. Operating rooms and staff are a scarce medical resource in their own fashion and must be allocated just as wisely as transplant organs [4]. Thus, OR managers, like other administrators, pursue efficiency over productivity when resources are limited.

Chernov’s data suggest that efficiency can be improved without impacting quality. Is this always true? Clearly not. In a reduction ad absurdum, one can imagine a system is so slapdash that many patients die on anesthesia induction, allowing for rapid room turnover and high throughput with minimal quality. Conversely, an untoward completely risk-averse obsession with procedural checks, rechecks, and safeguards can clearly slow down an OR system. However, neither of these extremes is likely to exist today in most generally well-managed OR systems and there seems likely to be room to improve both efficiency and quality somewhat in each. Indeed, it is interesting that Chernov et al.’s [1] actually reported a decrease in “events” with implementation of their management system, even though this was clearly not statistically significant.

The real question may be how to think about these issues. Current quality improvement methodology emphasizes outcome measurement, tracking and trending, and serial attempts to change the system while monitoring the outcomes to see if they improve. This is easy to do for an easily quantitated metric like overtime or turnover time, but harder to do for “quality” because quality is harder to define. Chernov and others often focus on frequencies of extreme events like perioperative death or failure to comply with protocols because these are measurable, but such extremes may not be sufficiently sensitive to smaller changes in quality.

Because OR efficiency can be measured using many metrics, balanced scorecard approaches are often used to reflect these different outcomes [5, 6]. However, the problem with scorecards is that when one element improves while another deteriorates, it may be difficult to decide how to weight these variables. We [7] previously applied data envelopment analysis (DEA) to the assessment of US Department of Veterans Affairs teaching hospital OR suites, encompassing not only classical OR productivity but also less traditional “productivity” metrics around teaching and research. The Department of Veterans Affairs NSQIP program [8, 9] has taught us over the past decades to reduce adverse outcome frequencies to observed to expected (O/E) ratios by comparing outcomes across a national patient sample using sophisticated risk adjustment methodology, and although benchmarking programs may not always improve OR efficiency substantially [10],¹ NSQIP has clearly improved VA surgical quality and a similar approach has yielded similar results in the private sector [11]. While some adverse surgical events represent postoperative system problems related to failure to rescue, the majority are either driven by patient characteristics which can be adjusted for or have their genesis in events and decisions in the OR suite. Perhaps it is time to adopt NSQIP-like methodology to more specifically define OR quality in a more sophisticated fashion that can be balanced against more traditional productivity and efficiency metrics, whether by scorecards or a DEA approach.

Declaration of Interest

The author reports no conflicts of interest. The author alone is responsible for the content and writing of the article.