Original Research A modified method for estimating volume–outcome relationships: application to percutaneous coronary intervention

Abstract

Objective: The objective of the current study was to propose an alternative method for measuring individual operator and peer volumes to use as predictors for adverse outcomes.

Study design: A retrospective analysis was performed to assess the volume–outcome relationship for percutaneous coronary intervention (PCI) performed in New York State between 1996 and 1999. This relationship was calculated using a modified method whereby physician volume was calculated using the previous year's volume, and hospital volume was calculated after subtracting the operator of interest's annual volume from the total. The primary outcome of interest was in-hospital mortality.

Results: Using the modified method, the odds ratio (OR) of in-hospital mortality was 0.74 (95% confidence interval (CI) 0.55–0.99; p=0.04) for cardiologists who performed 75–174 procedures annually and 0.80 (95% CI 0.61–1.04; p=0.1) for cardiologists who performed ≥175 procedures annually compared with the lowest-volume operators. With the conventional approach to volume measurement, no relationship between cardiologist volume and mortality was found. Patients who underwent PCI in hospitals where their physician's peers had an annual volume of 600–999 or ≥1,000 cases had a significantly reduced odds of mortality (OR = 0.73; 95% CI 0.57–0.92; p=0.01; and OR = 0.77; 95% CI 0.62–0.95; p=0.01) compared with patients treated by physicians with an annual peer volume of <600 cases. The conventional method did not detect any significant correlation between hospital volume and in-hospital mortality.

Conclusion: The alternative approach to measuring cardiologist and peer volumes proposed in this study leads to more precise estimates of volume–outcome relationships than the conventional approach.

Keywords::

• volume-outcome relationship
• percutaneous coronary intervention
• angioplasty
• mortality

Introduction

The relationship between the experience of healthcare providers, as measured by their volume of a particular procedure, and clinical outcomes has been studied extensively since the pioneering work of Luft et al1. Understanding the relationships between procedural volume and outcomes has important implications for all stakeholders in the healthcare system, including referring physicians, healthcare payers and patients. As a result, volume–outcome research has been performed for a large variety of medical and surgical procedures, including, but not limited to, percutaneous coronary intervention (PCI), coronary artery bypass grafting, abdominal aortic aneurysm repair, pancreatic resection and cystectomy2–18. A number of these studies have demonstrated that treatment success is associated with the volume of procedures performed by the physician and hospital2–15; other studies, however, have been less conclusive16–18.

Based on results from studies demonstrating an inverse relationship between PCI volume and adverse outcomes, the American College of Cardiology and the American Heart Association (ACC/AHA) recommended in 199319 that cardiologists should perform at least 75 procedures per year to maintain competence in PCI. However, to justify such recommendations, precise and reliable estimates of volume–outcome relationships must be obtained. Most previous studies have used analytical methods that have resulted in potentially imprecise and biased estimates of the relationship between volume and outcome. First, by relating physician volumes and outcomes within the same year, the conventional approach has related expertise acquired later in a given year to outcomes incurred earlier in the same year. Second, by assigning the same annual hospital procedure volume to all operators performing procedures in that year, the experience of an operator's peers is erroneously assumed to be identical for all operators.

The primary objective of the present study was to describe and test an alternative approach that measures individual operator and hospital PCI volumes prior to use as predictors for adverse outcomes. When the modified approach was compared with the conventional methods used in prior work important differences emerged.

Methods

Conventional method

Research on volume–outcome effects typically uses the following type of equation:

OUTCOME_it = α₀ + α₁MDVOL^jkt + α₂HOSPVOL_kt + θX_it + ϵ

where OUTCOME_it = clinical outcome for the ith patient (operated by the jth physician in the kth hospital) in year t; MDVOL_jkt = volume of the procedure performed by the jth physician in the kth hospital in year t; HOSPVOL_kt = volume of the procedure performed at the kth hospital in year t; X_it = a vector of the characteristics of the ith patient in year t; α₀–α₂ and θ = coefficients to be estimated; and ϵ = a randomly distributed error term.

Moreover, the conventional approach measures HOSPVOL as:

HOSPVOL_kt = ∑jMDVOL_jkt,j = 1,…n

Thus, according to these equations, physician volume is measured in the same year as the outcome, and hospital volume is measured as the sum of the individual volumes of all physicians in that hospital performing the procedure in question during a particular year.

Limitations of the conventional method

The conventional method has two important limitations. First, by measuring physician volumes and outcomes within the same year, the conventional method relates expertise acquired later in that year to outcomes incurred earlier in the same year. If physicians tend to perform the same amount of procedures in each year, the statistical bias associated with the conventional approach will be low. But with emerging technologies or newly trained operators, physician experience changes dramatically from year to year. In such cases, the bias from the conventional approach will be most severe.

The second limitation results from the second equation. As that equation indicates, hospital volume is typically measured as the sum of a given procedure performed by all physicians at a given hospital in a particular year. However, this approach confounds the experience of an operator's peers with the operator's own experience. The intercorrelation between individual physician and hospital volume may be substantial for high-volume physicians or for hospitals with few physicians performing the procedure. Such collinearity inflates the variance of the coefficient estimates, causing regression coefficients to be estimated imprecisely. Moreover, because peer effects have been measured inaccurately, one may anticipate biased estimates of the relationship between peer effects and outcomes.

Modified method

Each of the problems associated with the conventional approach, bias and imprecision, may be addressed by a straightforward modification of the current estimation approach. The first source of bias resulting from including current year measures of physician volume may be avoided by using the total number of procedures the physician performed in all hospitals during the prior year in order to predict the current year's outcomes. Moreover, this approach is consistent with the economic theory of ‘learning by doing’, which asserts that experience gained by individuals through repeating the same production process increases their productivity20. All the empirical studies exploring the relationship between accumulated experience/knowledge and productivity defined and measured experience gained prior to the outcome21–25. By using the same-year volume instead of the volume before a particular procedure, the concept of ‘accumulated experience’ that affects productivity in the theory may actually include experience acquired after that procedure. To illustrate, consider a physician who will perform 100 PCIs during the course of the current year, but who performed no such procedures during the previous year. For the physician's first patient who undergoes a PCI in the current year, the physician's experience is really 0, yet the conventional approach assigns the physician an experience of 100 procedures, which in fact is only acquired later in the year. Of course, it is also possible to construct cases where this method does not measure experience well. Nevertheless, the authors believe that their recommended approach is more appealing conceptually because it relates the physician's accrued experience to subsequent outcomes rather than the reverse.

A second rationale for the proposed methodology is a statistical one. The recommended approach avoids the problem of reverse-causation effects. Under the conventional approach, it is more difficult to infer which way the causation runs. For example, doctors who experience poor outcomes may end up performing fewer procedures during the course of the year. In other words, the outcomes a doctor experienced early in the year may affect the total number of procedures that doctor performs during the course of that year. For example, a string of poor outcomes may lead the doctor to cut back on procedures later in the year. In this case, the total volume of procedures performed in that year depends on the outcomes. In more technical terms, the conventional measure of volume is endogenous, because it is in part determined by the outcome. But the lagged value of volume is predetermined and as a result avoids these reverse-causation effects.

The wide use of current-year volume may in part reflect data constraints. One potential argument for using the current-year volume is if the current-year volume and prior-year volume are highly correlated. However, as the author's data show, this is not necessarily the case. Indeed, the volumes of many of the operators in this data vary significantly from year to year. This will also be true for new treatments and technologies where operator experience is changing rapidly from year to year. In the present study the recommended method will be compared with results obtained using the conventional approach.

The second problem associated with peer effects may be rectified by omitting a physician's own experience in calculating hospital volume. This method better reflects differences in the expertise available from the operator's peers. One argument for using total hospital volume rather than this peer measure is that total hospital volume reflects the experience of supporting units such as physician assistants and nursing staff. Nonetheless, the authors strongly prefer their measure for capturing peer effects, both because it is a more precise measurement of these effects and because it avoids statistical problems associated with including total hospital volume with individual operator volume. The experience of supporting units may be important in terms of patient satisfaction and reducing physician's burden and may thus be an additional relevant factor to include in a model. But if one is interested in capturing peer effects, the authors believe that their approach provides a more accurate measure.

Interactions between volumes of individual operators and institutions were also estimated, the latter measured by the individual physician's peer volume. Such synergies between the two experience measures may be quite important, as demonstrated in several studies2,26. However, given the collinearity between individual physician volume and total hospital volume as well as the use of current-year volumes to predict outcomes, the precision and reliability of these estimates are questionable. Hence, such interaction models were re-examined to test the hypothesis that having high-volume peers improves the performance of low-volume operators.

Limitations of the modified method

Whilst the modified approach has certain advantages as described above, it too has limitations. First and most obviously, it requires more data. One must have at least 2 years of data on each physician in order to relate lagged volumes to current outcomes. Owing to this limitation, one cannot include physicians who moved into a state during the current year, as information on their previous year's volume will generally be unavailable. A third limitation is that the modified method also has some measurement error. Ideally, one would like to measure physician volume up to and including the last procedure performed. But the modified approach includes last year's experience, not, for example, last week's.

Ultimately, whether the proposed method is superior is an empirical issue. If it leads to more precise estimates of physician and peer volume and a better overall model fits, the approach has merit. On the other hand, if it leads to little change in the model, or to worse performance, then there would be little motivation for using it. The two approaches are compared in the empirical analyses below.

Data and variables

To make the study comparable with previous PCI volume–outcome research, the Coronary Angioplasty Reporting System (CARS) data from the Department of Health of New York State during the period 1996–1999 were used. The 1996 data are only used to calculate the lag volume for 1997. The CARS was established in 1991 by the New York State Department of Health and its Cardiac Advisory Committee. The registry consists of all PCIs performed in New York State and thus constitutes a large, multicentre, mandatory, quality-controlled PCI database.

The system collects prospectively defined data elements for each patient undergoing PCI in New York State, including information on demographics, co-morbid conditions, patient, operator and hospital identifiers, pre-procedural risk factors, procedural details, complications, in-hospital outcomes and discharge status from the hospital, as well as provider (hospital and operator) identifiers. All the analyses are based on procedures performed from 1997 through to 1999.

Outcome measures

The primary outcome of interest is in-hospital mortality.

Physician volume

Cardiologists who performed PCI are identified by unique assigned operator codes. It was, therefore, possible to calculate the annual volume of PCIs performed by each cardiologist. The cardiologists are grouped into three categories according to their annual volumes in the year prior to the year of interest (1-year lag volume) in the modified method and to the procedure total for the year of interest in the conventional method. The three volume categories are defined as low volume (<75 procedures/year), medium volume (75–174 procedures/year) and high volume (≥175 procedures/year).

The authors acknowledge that different volume thresholds might significantly affect the estimation of volume–outcome relationships. The thresholds used in this study are based on those used by Hannan et al2 so as to be consistent with earlier work that used the conventional volume–outcome estimation approach. The choice of volume cut-offs was also motivated so as to coincide with guidelines for minimum acceptable annual PCI volumes of 75 for cardiologists and 600 for hospitals set by the ACC/AHA guideline19.

Peer volume

Peer effects are measured for each cardiologist as the number of PCIs performed by other physicians at the hospital in a given year. The peer volumes used in the modified method are lagged 1 year; the hospital volumes used in the conventional approach are not. Because many cardiologists performed PCIs in multiple hospitals, the 1-year lag peer volume for each operator was calculated using the volume at the hospital where the most procedures were performed in the year of interest, minus their own volume of the previous year. The hospitals are then categorised into three groups according to the 1-year lag peer PCI volumes: <600; 600–999; and ≥1000. The new system puts more cardiologists in lower peer-volume groups. For example, in 1997 there were 66, 68 and 70 cardiologists in the low, mid and high hospital volume groups, respectively. When classified by peer volume, there were 81, 67 and 56 cardiologists in the low, mid and high peer volume groups, respectively. Each size group is still very well represented.

Statistical analysis

Data were extracted using SAS statistical software, version 9.1 (SAS Institute Inc., Cary, NC). Multivariate analyses were then performed using STATA statistical software, version 8.0 (StataCorp LP, College Station, TX). The statistical analysis includes two types of models. First, the effect of operator volume and hospital/peer volume on hospital mortality was estimated. Dummy variables were constructed for the cardiologist and hospital/peer volume in each volume category. A second set of analyses added dummy variables for the interaction between cardiologist and peer volume to see whether the mortality effects of individual physician volume varied across high- and low-volume hospitals.

In addition to indicators of physician and peer volume, all analyses adjusted for patient co-morbidities such as diabetes and congestive heart failure, clinical risk factors, demographic characteristics such as age, race and gender, and year indicators.

Results

The mean age of the 95,564 patients was 63.4 years. The majority of patients were male and White. The overall in-hospital mortality was 0.85% (Table 1). Table 2 presents the number of hospitals and cardiologists in various volume ranges according to their annual PCI volumes. There were 33 hospitals that performed PCI in New York State during 1996 and 1997 and 34 hospitals in 1998 and 1999. The number of cardiologists who performed in New York State grew from 195 in 1996 to 233 in 1999, an increase of 19%. The number of patients who underwent PCI rose 39%, from 25,654 in 1996 to 35,738 in 1999.

Table 1.  Baseline characteristics of patients undergoing percutaneous coronary intervention in New York State, 1997–1999.

Demographics	n=95,564
Mean age (years)	63.37
Female (%)	32.15
Black (%)	6
Cardiovascular history (%)
Open heart surgery	16.02
Previous heart failure	5
Heart failure this admission	5.80
Diabetes	23.66
Vascular disease	8.30
Outcomes
In-hospital mortality (%)	0.85
Emergency CABG (%)	0.47
Length of stay (days)	3.78

CABG, coronary artery bypass grafting.

Table 2.  Numbers of hospitals and cardiologists by year and by PCI volume range groups.

	1996	1997	1998	1999
No. of hospitals (by PCIs/year)
<600	14	12	11	10
600–999	10	11	9	9
≥1000	9	10	14	15
Total no. of hospitals	33	33	34	34
Total procedures	25,654	29,516	33,310	35,738
Mean hospital volume	777.39	894.42	979.71	1,051.12
No. of cardiologists (by PCIs/year)
<75	75	71	69	71
75–174	69	69	77	81
≥175	51	65	71	81
Total no. of cardiologists	195	205	217	233
Mean cardiologist volume	131.56	143.98	153.50	153.38

PCI, percutaneous coronary intervention.

Between 1996 and 1999, not only did the number of cardiologists and hospitals performing PCI increase, but the average number of procedures per cardiologist and hospital also rose. For hospitals, the average annual volume in 1996 was 777, increasing to 1,051 by 1999. The number of hospitals performing ≥1,000 procedures annually increased from 9 to 15 during the same time period. For cardiologists, the average annual volume grew from 132 in 1996 to 153 in 1999. The number of cardiologists who performed ≥175 procedures annually also increased, from 51 to 81.

Volume–outcome relationships

The modified method demonstrates that high-volume cardiologists have significantly lower mortality than low-volume providers who perform <75 procedures annually (Table 3). Relative to these low-volume physicians, the odds ratio (OR) of mortality is 0.74 (95% confidence interval (CI) 0.55–0.99; p=0.04) for cardiologists who performed 75–174 procedures annually and 0.80 (95% CI 0.61–1.04; p=0.1) for cardiologists who performed ≥175 procedures annually. In contrast, using the conventional approach to volume measurement, no statistically significant relationship between cardiologist volume and mortality is demonstrated for medium-volume (OR 0.93; 95% CI 0.65–1.33; p=0.7) or high-volume (OR 0.86; 95% CI 0.6–1.22; p=0.4) operators. Similarly, the modified approach demonstrates that peer effects are important to consider. In particular, patients who underwent PCI in hospitals where their physician's peers had an annual volume between 600 and 999 experienced a significant reduction in the odds of mortality (OR 0.73; 95% CI 0.57–0.92; p=0.01). Patients undergoing PCI in the highest peer-volume hospitals (≥1,000) also had significantly reduced odds of mortality (OR 0.77; 95% CI 0.62–0.95; p=0.01). In contrast, with the conventional calculation no significant relationship between hospital volumes and in-hospital mortality is observed (Table 3). The modified model also performed slightly better in terms of overall model fit, as measured by pseudo r-square values and likelihood ratio tests.

Table 3.  Volume effects for in-hospital mortality: comparison between conventional and modified methods.

	Range	Conventional method			Modified method
		OR	p-value	95% CI	OR	p-value	95% CI
Cardiologist volume	<75	1			1
	75–174	0.93	0.7	0.65–1.33	0.74	0.04	0.55–0.99
	≥175	0.86	0.4	0.6–1.22	0.80	0.1	0.61–1.04
Peer/hospital volume	<600	1			1
	600–999	0.98	0.92	0.71–1.37	0.73	0.01	0.57–0.92
	≥1000	0.78	0.11	0.58–1.06	0.77	0.01	0.62–0.95

OR, odds ratio; CI, confidence interval.

Operator and institutional volume interaction

Table 4 presents the interaction between volume of individual cardiologists and their peer volume in the same hospital using the modified method. For in-hospital mortality, compared with the outcome for lowest-volume cardiologists (<75) with the lowest peer volumes (<600), all interactions between other cardiologist volumes and peer volumes incur less mortality. Particularly, patients who underwent PCI by medium-volume cardiologists (75–174) with the highest peer volumes (≥1000) were 61% less likely to die (OR 0.39; 95% CI 0.24–0.64; p<0.001). The highest-volume cardiologists (≥175) with medium peer volumes (-600–999) also incurred significantly less mortality (OR 0.49; 95% CI 0.32–0.75; p=0.001). The highest-volume cardiologists with the highest peer volumes had an OR of 0.64 of incurring mortality (95% CI 0.44–0.94; p<0.05).

Table 5 compares the interaction of operator and institutional volume as demonstrated by the conventional and modified approaches. In contrast to the modified method, the conventional approach shows no significant interaction between physician and hospital volume.

Table 4.  Interaction between individual and peer volume for in-hospital mortality: modified method.

Cardiologist volume		Peer volume
		<600	600–999	≥1,000
<75	Risk-adjusted OR	1	0.63	0.79
	95% CI		0.32–1.23	0.46–1.36
75–174	Risk-adjusted OR	0.77	0.70	0.39
	95% CI	0.49–1.21	0.44–1.1	0.24–0.64
≥175	Risk-adjusted OR	0.74	0.49	0.64
	95% CI	0.49–1.13	0.32–0.75	0.44–0.94

OR, odds ratio; CI, confidence interval.

Table 5.  Interaction between operator volume and peer/hospital volume for in-hospital mortality: conventional vs. modified method.

Modified method	Range		Peer volume
			<600	600–999	≥1,000
Lagged cardiologist volume	<75	Risk-adjusted OR	1	0.63	0.79
		95% CI		0.32–1.23	0.46–1.36
	75–174	Risk-adjusted OR	0.77	0.7	0.39
		95% CI	0.49–1.21	0.44–1.1	0.24–0.64
	≥175	Risk-adjusted OR	0.74	0.49	0.64
		95% CI	0.49–1.13	0.32–0.75	0.44–0.94
Conventional method	Range		Hospital volume
			<600	600–999	≥1,000
Current-year cardiologist volume	<75	Risk-adjusted OR	1	0.91	0.97
		95% CI		0.34–2.46	0.49–1.93
	75–174	Risk-adjusted OR	1.03	0.97	0.79
		95% CI	0.57–1.87	0.54–1.75	0.45–1.37
	≥175	Risk-adjusted OR	0.97	0.94	0.72
		95% CI	0.45–2.1	0.54–1.62	0.43–1.21

OR, odds ratio; CI, confidence interval.

Comment

Performance or treatment of a high volume of procedures or medical conditions has long been associated with better outcomes. However, the methodological rigor of these assessments has only infrequently been addressed. Prior volume–outcome studies have been criticised for several reasons: failure to identify the simultaneous contribution of hospital and physician volumes to outcomes; suboptimal risk adjustment; and arbitrary definitions of high and low volumes2. The current study adds to prior literature by suggesting that the commonly employed methods of defining physician and hospital volume introduce bias and imprecision into the analysis of the relationship between volume and outcome.

Conventional methods for calculating physician volume will be accurate only if physician volume varies little from year to year. If that were the case, there would be little gain from using a lag volume measure rather than a concurrent one. But with new practitioners entering the field and with new procedures, volumes are likely to vary widely from year to year. To illustrate, consider that approximately 273 interventional cardiologists are trained each year in the US and begin their first year of practice with 0 accumulated procedures. If a newly trained interventional cardiologist performs 200 procedures during the first year of practice, the experience on their very first case and all subsequent cases throughout that year is assigned a value of 200. Clearly this method inflates the experience of a new operator, whereas with the modified approach a new operator would be assigned an experience based on the 1-year lag volume of 0.

Hospital volume has historically been assumed to be a surrogate for the experience of an operator's peers. Thus, an experienced cohort of peer physicians could negate some of the detrimental effect of low volume for a particular operator by allowing ready access to physicians with enhanced judgment and advanced technical skills. Conventional methods have included the operator's own volume in the calculation of annual hospital volume4–6. However, this approach confounds the experience of an operator's peers with the operator's own experience. For example, consider a hospital that performs 1,000 procedures in a year. Operator 1 performs 700 of these procedures, operator 2 performs 250 and operator 3 performs 50. Under the conventional approach, hospital volume would be the same for each physician (1,000), implying that the peer effects would be the same. However, in actuality, the peer effect for would be 300 for operator 1, 750 for operator 2 and 950 for operator 3. Clearly, operator 1 has the least amount of expertise on which to draw from his peers whilst operator 3 has the greatest, differences that are captured by the modified approach.

Prior studies have suggested that physician and hospital volumes should not be judged in isolation but rather that the interaction of physician and hospital volume should be considered in recommending volume thresholds for physicians3. The modified method using 1-year lag volumes and peer volumes instead of hospital volumes confirms the contribution of experienced peers to the outcomes of patients undergoing PCI.

Healthcare consumer organisations such as the Leapfrog Group recognise procedural volume as an indicator of improved outcomes27. As a result, healthcare consumers are directed to high-volume hospitals for certain procedures, including PCI. Nevertheless, over the past three decades of volume–outcome research, the methodology for assessing the relationship between volume and outcomes has remained unchanged. The data presented in the current study suggest the need to re-assess the methods for calculating physician and hospital volumes to obtain more precise estimates of the volume–outcome relationship.

Limitations

As argued earlier, using the 1-year lag volume is an improvement over the existing approach of using concurrent volume. Ideally, however, one would like a measure that includes the physician's years of experience with the procedure and properly discounted career overall volume of such procedures performed. This would require a longitudinal data set sufficiently long to allow us to trace each physician's total experience with the procedure. In future work, longer time periods need to be examined to determine how the physician's total prior experience affects outcomes.

A second limitation is the choice of volume thresholds as previously discussed. Further research is needed to detect the thresholds using these proposed measures for volumes.

Conclusion

This study demonstrates that the proposed modified method for measuring individual operator and peer volumes led to substantively different estimates of volume–outcomes relationships than work using the conventional approach of concurrent volumes and hospital volumes. Using a modified method, the study found that hospital mortality from PCI is significantly higher among low-volume operators, whereas the conventional approach fails to detect this difference.

From a policy perspective, these findings are important because they suggest that PCI by low-volume operators continues to have adverse quality implications. This, in turn, lends more support for ACC/AHA guidelines calling for cardiologists to perform at least 75 PCIs annually. A useful direction for further research is to test this approach against the conventional one on different clinical databases to assess volume–outcome relationships, not only for PCI but for other medical services and procedures as well.

Acknowledgements

Declaration of interest: The authors have declared no conflict of interest and have received no payment for the preparation of this manuscript. The views represented in this paper are those of the author's and may not represent those of the New York State Department of Health or its Cardiac Advisory Committee.