Abstract
Introduction. In health care, measures of performance are needed at producer level for improving the treatment processes and at system level for steering purposes. In addition, measures that enable reliable comparisons of producers with respect to each other should encourage them to develop their treatment processes to attain better positioning in benchmarking.
Methods. The main innovation of the Performance, Effectiveness, and Costs of Treatment episodes (PERFECT) project is to measure performance using existing linkable information available from registers within well-defined care episodes in a whole population. Finnish health care and related registers are used for constructing the disease-specific databases, with rich content on treatment processes and complete follow-up data.
Results. The PERFECT project has developed numerous performance indicators that can be used to evaluate health policy actions as well as to create regional and hospital-level benchmarking data. In PERFECT, the idea is to eliminate individual-level variation from the performance indicators by using individual-level data and proper risk adjustment methods. The focus of our interest is in the variation at the producer or regional level.
Conclusions. Our experience shows that the utilization of population-level health care registers with an episode-of-care approach enables a continual system and producer-level performance measurement.
| Abbreviations | ||
| AMI | = | acute myocardial infarction |
| ATC | = | Anatomical Therapeutic Chemical classification |
| DRG | = | diagnosis-related group |
| ICD-9 | = | International Classification of Diseases, ninth version |
| ICD-10 | = | International Classification of Diseases, tenth version |
| PERFECT | = | Performance, Effectiveness, and Costs of Treatment episodes |
| SII | = | Social Insurance Institution, Finland |
| THL | = | National Institute for Health and Welfare |
| Episode of care | = | The entire treatment pattern from the beginning (e.g. the acute stage) of the disease to the end of the treatment over any organizational boundaries to solve the health problem at hand in a specific time frame |
| Index admission | = | The hospital admission starting the episode |
| Index day | = | Admission day of the index admission |
| First hospital episode | = | Hospital treatment beginning on the index day and terminating on the first discharge home, death, or after a specified time of continuous inpatient care |
| Total episode of care | = | All the health service use that emerged during the follow-up period |
Key message
The utilization of population-level health care registers with an episode-of-care approach enables a continual system and producer-level performance measurement.
Introduction
A desirable health care performance measure is one that reliably and accurately reflects the process, costs and outcome of the care provided by providers. Such measures give producers valuable information for improving their treatment processes and, at system level, for steering purposes (resource allocation). In addition, measures that enable reliable comparisons of producers with respect to each other should encourage them to develop their treatment processes to attain better positioning in benchmarking.
A performance measure of a producer must be carefully constructed and comparable in order to be useful. There are many important and well-documented methodological and practical questions that need to be considered when administrative data is used for performance measurement (Citation1–4). Indicators will vary due to provider or regional-level, individual-level and random variation. In Performance, Effectiveness, and Costs of Treatment episodes (PERFECT), the idea is to eliminate individual-level variation by using individual-level data and proper risk-adjustment methods. The focus of our interest is in variation at producer or regional level.
The main innovation of the PERFECT project is to measure performance using existing linkable information available from registers within well-defined care episodes in a whole population. This is possible in Finland since there is a rich research tradition and the registers cover the whole population (Citation5). The quality of the registers has been shown to be excellent (Citation6–14). Sund (Citation15) has assessed the validity of the data against performance assessment purposes in the case of hip fracture.
In addition, the records in different registers are deterministically linkable for all individuals on the basis of personal identification code. Thus, it is possible to follow patients to track their use of services – not only in specialized health care, but also in the sectors closely related (care of the elderly, primary inpatient care, purchases of prescribed drugs). When using services people leave traces in these data and they are recorded upon decease in the death statistics.
Implementing register-based performance evaluation requires – in addition to availability of comprehensive data – methodological understanding and a multidisciplinary approach (Citation15). Health system knowledge is essential for deciding the scope and the specific questions to be addressed, and this needs to be complimented with understanding of the possibilities and limitations of register information. It is important to know how registers can be integrated. Clinical expertise is needed when appraising details of the management of a disease, and economic, statistical and data mining knowledge are required to ensure that the methodology is appropriate. Finally, all the aforementioned must be integrated during the entire process. In PERFECT, the performance indicators have been developed by separate expert groups, whose members (some 50) are leading clinical experts on the diseases as well as experts in health economics and statistics.
The main aim of the PERFECT project is to develop performance indicators that can be used to evaluate health policy actions as well as to create regional and hospital-level benchmarking data that allow decision-makers to learn from best practices. This article describes the definitions and data processes and the methodological choices we have made in the project.
Episode of care
In PERFECT, the focus is on individuals and the entire treatment chain of their health problem. Inferences about performance (effectiveness) are then drawn at the meso level. The project combines a microeconomic disease-based strategy (Citation16) with an episode-of-care approach.
The concept of an episode has been used to distinguish between discharge and single intervention. The idea of an episode approach is not new (Citation17), but the implementation of this concept has been challenging in practice (Citation18) and has not been done before on such a large scale.
An episode of care refers to the entire treatment pattern from the beginning (e.g. acute stage) of the disease to the end of the treatment over any organizational boundaries to solve the health problem at hand in a specific time frame. Thus the protocol for an episode includes definition of start and finish dates (follow-up time) as well as inclusion and exclusion criteria, which are used as rules in constructing a comparison data set for a specific disease group, for which all indicators are calculated.
In PERFECT, follow-up data for at least one year is available for each patient. The main observable events in the register data are admissions, operations and discharges as well as deaths. The secondary observable events are outpatient visits and medication purchases. In addition to the follow-up data, we have similar information on the service use history of the patients. Using the available data it is possible to reconstruct treatment pathways that describe what has happened before and after an operation on a daily basis (Citation19). It is noteworthy that in Finland the discharge data is actually specialty discharge data and therefore provides a more accurate view of the events within an episode than is the case for mere hospital discharge data ().
Figure 1. An example of events within an episode of care.
Episode of care provides a framework that can be operationalized in terms of linkable register data. In order to model the episode of care and calculate the measures of performance, the nature of the disease and the characteristics of the patients must be taken into account, since they are the key factors affecting the treatment decisions and implementation of care. For example, the onset of a disease may be known accurately (e.g. hip fracture) or it may be practically impossible to define the start of the disease (e.g.. arthritis). Similarly the end of an episode of differs.
The hospital admission starting the episode is in our terminology the index admission, and the admission day the index day. The first hospital episode is defined as beginning on the index day and terminating on the first discharge home, death, or after a specified time of continuous inpatient care depending on the disease. The first hospital episode describes the acute phase of treatment. The follow-up period ends one year after the index day or at death, whichever occurs first. The total episode of care includes all the health service use that emerged during the follow-up period.
Registers and the construction of the database
The primary national health registers () were selected for the PERFECT study. The ethical committee of the National Institute for Health and Welfare (THL) has approved the study and the register authorities have permitted the linking of register data for the purposes of this study. All the register material is acquired annually from the data governing institutions, as data accumulates continuously and in PERFECT the databases are updated and expanded accordingly. The databases of each sub-study are then formed separately after the original register material has been gathered. At every step close attention is paid to the accuracy of data processing and especially to protection of patient privacy. When working with many registers in many different sub-projects data management becomes crucial.
Table I. The Finnish health registers used in the PERFECT sub-studies.
The clinical aspects of the disease-specific protocols for forming the comparison databases are explained in more detail in the disease-specific articles in this supplement. The general stages of data processing are i) to define the patient population, ii) to collect the register material for the patient population at hand, iii) to define the index discharge for the patients (the start of the episode) and form the necessary variables concerning the care given, iv) to check the service use history (and follow-up) of the patients in order to form the patient history and outcome variables, v) to form the comorbidity variables for the patients, vi) to calculate the direct costs of care and finally, vii) to combine the information of the previous stages to form the comparison database.
Indicators
The implementation of the benchmarking function of the project is carried out through basic reports (available on the internet) which include performance indicators on the content of care, costs and outcomes between 1998 and 2008. The content of the basic reports is tailored to meet the needs of each disease, and they are presented on two levels:
I) By Hospital Districts (organizations responsible for providing specialist care in Finland), based on the municipality of the patient. In the most populated Hospital District (Helsinki and Uusimaa) we have calculated indicators also by so-called hospital care directs, which are local areas responsible for care under the hospital district.
II) By hospitals, based on the patients treated in a hospital. In the reporting we present indicators only for hospitals that have annually treated at least 50 patients included in the comparison data. Both public and private hospitals are included. The patients are followed over organizations, for example, in hip replacements the hospital of the primary operation is discredited in the event that their patient has revision surgery for the same joint at another hospital later.
Baseline statistics
In each sub-project, the baseline statistics of the patient population in each meso-level unit are given. These include, for example the number of patients, mean age and proportion of males and females.
Process and cost indicators
The process indicators describe health care service use during the episode of care. The indicators include measures such as length of stay in hospital (in initial hospitalization and during the follow-up period), procedures and other treatment practice, and use of medicines. The cost indicators usually describe the cost of first hospitalization as well as the total cost per person in a given follow-up period (usually one year).
The registers in Finland do not contain the exact individual-level cost information of each treatment and stay. The estimation of costs is based on the use of different types of services in the follow-up period. We apply both an episode-based approach and a person-based approach (see e.g. (Citation20)) in order estimate direct health care costs while indirect costs have been omitted. Our costing method does not provide accurate information on the actual (true) costs of individual patients, but a measure of the average amount of resources used in the treatment of a certain disease. In addition, using multiple registers enables patients to be tracked with respect to any care that they have received before and after the onset of the disease, i.e. the pathways of the patients in the health care system are precisely traceable. Only primary outpatient care lacks a national register (which will be provided in 2011).
As a general method, the discharges were costed based on their diagnosis-related group (DRG). For such care that is not classifiable in any DRG, national unit prices were used (Citation21). The costs of prescribed medicine were obtained from the registers of the Social Insurance Institution (SII).
In some cases, DRG was not considered to be appropriate for cost measuring. For example, in total hip and knee replacements there is only one DRG for both hip and knee primary arthroplasties. In such cases we applied a modelling strategy for the discharge. Using individual-level cost accounting data from one hospital district we constructed a model which we then used to predict the costs of the discharge. The model includes information about procedures, duration, discharge status (home, another institution, death), and other various disease-specific variables. Besides hip and knee replacements, this approach was applied to the treatment episodes of hip fractures, very low birth weight infants and acute myocardial infarction.
Outcome indicators
Here we understand by the term outcome those measures of health improvements (or decrease in deterioration) attributable to health care. Ideally we are interested in the outcome in real life in terms of changes in health-related quality of life as well as in survival. However, health-related quality of life is not routinely available and thus cannot be used for comparisons between regions or hospitals.
The outcome measures used in PERFECT can be categorized into measures of mortality (by which we mean the proportion of individuals who die from any cause within a certain period after the start of follow-up, such as one-year mortality after a hip fracture) and into measures derived from health service use (i.e. readmission) ().
Table II. Outcome measures of the PERFECT project in five patient populations.
After the patient has been discharged and the initial (acute) treatment has ended, patient follow-up starts. During the follow-up the patient might develop a specific complication (e.g. post-surgery infection, post-arthroplasty dislocation) or s/he might be hospitalized because of a wider set of symptoms. The discharge register, combined with the other registers, offers a means of obtaining information about the individual’s condition (the level of care) on a daily basis. For example, in hip fractures this information has been developed into diagrams depicting the treatment patterns and incorporating thousands of indicators in a single figure (Citation22).
Risk adjustment
When comparing regions, hospitals and years, patient-associated factors must be accounted for. We have endeavoured to ensure meaningful comparisons using three steps. Firstly, we have defined the patient groups so that they are as comparable as possible. The disease-specific protocols for episodes exclude some patients from the comparison data (see tables of data flows in the disease-specific sections). For example, the acute myocardial infarction (AMI) comparison data includes only hospitalized AMI patients aged 40–85, and excludes patients hospitalized for AMI during the previous year (365 days) and those institutionalized before the index hospital admission (Citation23). Secondly, we have gathered information on risk factors from the patients’ medical history. Thirdly, we have applied statistical models to adjust the indicators and calculated the confidence intervals.
Measurement of risk factors
Numerous comorbidity measures are available when using administrative data (for a review see e.g. (Citation24–26)), the most common being the Elixhauser method (Citation27) and the Charlson Comorbidity Index (Citation28). Besides comorbidity chartings based on hospital discharge data, there are developments based on the Anatomical Therapeutic Chemical (ATC) classification system (Citation29–31). Using experience of these measures, data availability as well as some statistical testing our disease specific expert groups separately tailored for each disease a set of variables, which were used for risk adjustment.
Three different databases are used to search medical events in patients’ records: the hospital discharge register, the register of special reimbursements, and the register of prescribed medicines. The use of medicine reimbursements is a specific Finnish feature. The various sources overlap, that is a person hospitalized for a specific comorbidity has a high probability of having been granted a special reimbursement for the comorbidity or of having purchased a prescribed medicine for that illness. On the other hand, the registers complement each other so that the general view of the comorbidities of patients is sharpened.
The main diagnoses of patients’ inpatient hospital treatments since the beginning of 1987 are checked for the conditions presented in . The assumption is that if a person has been in a hospitalized with a main diagnosis of a comorbidity before the start of the episode, s/he has had the comorbidity at the time of the event under study. Hospital discharges for the event or afterwards are not checked for comorbidities. In Finland, there are no separate fields in the discharge register for either complications or for comorbidities, and the coding of secondary diagnoses is known to be poor (Citation32). The special reimbursements are examined for each individual from the beginning of 1995 until the event. Similarly, the purchases of prescribed medicine are checked, but only for the last 365 days before the start of the episode of care.
Table III. The definitions of comorbidities in PERFECT according to register source.
The criteria for comorbidities (including International Classification of Diseases, ninth and tenth version (ICD-9 and ICD-10) classifications, Anatomical Therapeutic Chemical (ATC) classification, and special reimbursements) have been developed in the project in collaboration with medical experts in each disease group and with administrative data professionals. The comorbidities in the project are presented in . The comorbidities used in each sub-study vary; the table provides a broad perspective on the comorbidities selected in the study and in the marking criteria.
In addition to the comorbid diseases listed in , a variety of conditions has been taken into account in the risk adjustment of the indicators. Death during follow-up affects many of the response variables (accumulated costs, treatment days, purchases of prescribed medicines). Institutional treatment hinders patients from purchasing prescribed medicines. Occurrence of and time since previous similar event (e.g. in AMI) and time since some other event of importance (e.g. time since previous stroke in AMI), status immediately before the event of interest (did the patient come from home or from, e.g. hospital) are other examples of conditions that may affect the treatment choices and the outcomes for patients. These factors have therefore been controlled for when applicable.
Methods of risk adjustment in PERFECT
There are many possible methods that could be used for risk adjustment, such as methods related both to observable confounders (standardization using different approaches, such as nonlinear regressions, propensity score, confidence intervals using shrinkage estimators and other Bayesian methods) and unobservable confounders (instrumental variable methods and two-stage methods) (Citation33–45).
In practise, the selection of appropriate methods is based on balancing what can be done on a routine basis with scientific and methodological aspects. As the number of indicators in PERFECT is high, the development of refined statistical models for each indicator separately is not feasible given the resources. In addition, the indicators are produced annually, so the indicator-specific methods ought to be updated annually.
Thus, we have chosen to carry out indirect standardization for measures of incidence, and for all other indicators a modelling strategy is adopted: logistic regression for dichotomic responses, and generalized linear model (log link, with gamma distribution for continuous variables and negative binomial distribution for discrete variables), and a two- or three-stage modelling strategy using the aforementioned models for medication use and costs, respectively. The simplified approach chosen in PERFECT is justified on practical grounds. In addition, our methodology is accessible to a wider audience than are more complex alternatives. We recognize that more advanced methods exist and that the reporting might benefit from those.
With modelling, the parameter estimates for the confounding factors are first estimated with the broadest possible data (i.e. all cohorts of the database), and then the predicted values of the dependent variable (probability in logistic regression) are summed to meso-level. The ratio of the sums of the observed values and the expected values of the dependent variable in the comparable unit constitute the risk-adjusted indicator.
The case of a dichotomous outcome variable
Many of the outcome measures in PERFECT are dichotomous (mortality, readmission etc.) and the interest lies in the probability of the event. A straightforward way of forming predictions for response probabilities is to use logistic regression as suggested, for example, by Ash et al. (Citation46). The predicted probability of an event for the ith case (PREDi) is calculated from the relationship
where Xij is the value of the jth characteristics for the i th individual, after fitting a logistic regression model to the individual level data. Solving for the individual predicted probability for the event gives
Then, for the meso-level unit k, the predicted individual probabilities of its patients are summed to form the expected value Ek. The indicator of the outcome for the unit is simply the ratio of observed (Ok) and expected values of the response variable:
Other models
Two- and three part models are applied for medicine purchases and cost of medicine purchases, respectively. In the first part, the probability of being discharged alive is predicted for every patient, and in the second part the probability of purchasing drugs is predicted for every patient discharged alive. The expected number of patients who bought drugs is the product of the two. For the medicine cost indicator, the third part is a generalized linear model with gamma distribution and log-link to predict the costs of medicines for those who had purchased drugs.
Conclusions
The PERFECT project might be the first application of register-based performance evaluation that has been published and it is being updated annually on a national level. The framework of the project is easily adopted for different settings (health care environments), the bottleneck perhaps being data availability and reliability. On a smaller scale (regions within a country, a set of providers, etc.) the difficulties of the method could be avoided and meaningful comparisons for steering purposes done.
The strength of the PERFECT study lies in the reliable nationwide register data. Population level data from all producers of specialized health care that is linkable individually is utilized in the study. In addition, the specialized health care data is complemented with data on care of the elderly, prescribed medicines and death statistics. The approach succeeds in considering the whole treatment chain, as patients’ service use is tracked across the units in the follow-up. In many other healthcare systems there is no such possibility and studies are often restricted to a single intervention.
In the hospital discharge register, coding of the main diagnoses and main procedures is excellent (Citation47). Episodes of care in the data can therefore be defined. Unfortunately, there are deficiencies in the coding of secondary diagnoses and minor procedures (Citation32). This affects the reliability of the process and outcome indicators. As the special reimbursement data and the purchases of prescribed medicines are used in defining the comorbid diseases, the effect there is smaller. Our approach of scrutinizing the medical history is an advantage in the sense that events occurring during the same hospitalization reflect both complications of care and comorbidity. The method enables a more reliable extraction of comorbidities.
The implementation of episodes of care to the analysis is both a challenge and a virtue for the study. The episodes need to be carefully defined for every disease in collaboration with clinical experts on the disease. Thus, the episodes vary with the diseases. In the case of hip replacements, for example, the progress of the underlying disease (arthritis) is unknown; we only observe the procedure but have no information on patient health status before and after treatment. For example in AMI, stroke, hip fracture and low birth weight infants the occurrence of the condition is rather clear-cut and the measurement of the outcome can be based on risk-adjusted health status after the index hospitalization. In addition to having information on the procedures and diagnoses during entire episodes, we have managed to approximate the economic effects of the episodes. Although the costs do not reflect the differences in producers’ unit prices, they do convey information on resource use between units and on different treatment patterns.
To ensure justifiable comparisons of producers, we have adjusted the indicators for the patient characteristics and comorbidities. For practical reasons (over 200 indicators) we have streamlined the methodology, knowing that more advanced methods for risk adjustment exist. Our experience is that risk adjustment is crucial for comparisons, and due to patient selection the effect is greater at hospital level than at regional level. In the supplementary tables we present annual trends as well as regional differences for a number of key indicators of the project. The regional indicators are calculated as three-year moving averages, since one-year figures are subject to random variation. The effect of risk adjustment varies between different indicators and diseases. For example, risk adjustment has a considerable effect on the annual trends of outcome indicators in AMI and hip fracture, but a negligible one in stroke and hip and knee arthroplasty. Among stroke patients, however, the risk adjustment has a considerable effect on outcome indicators for hospital-level comparisons (Citation48).
The utilization of population-level health care registers with an episode-of-care approach enables continuous system and producer-level performance measurement. The health system registers in Finland have made this evaluation possible, but the registers have to be further developed in many respects to enable even better evaluation of the health care system. In the future, the challenge will be to incorporate primary care outpatient visit information in the framework. In addition, the adopted methodology will be expanded to other diseases.
Supplementary Tables S1–S50
Download PDF (202.7 KB)Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
Funding/Support: Research in the database has been supported by unrestricted grants from the Finnish Academy and Finnish Innovation Fund SITRA.
Role of the sponsors: The funding sources for this study had no role in the design and conduct of the study; in the collection, management, analysis and interpretation of the data; or in the preparation, review or approval of the manuscript.
Related Research Data
References
- Wray NP, Ashton CM, Kuykendall DH, Hollingsworth JC. Using administrative databases to evaluate the quality of medical care: A conceptual framework. Soc Sci Med. 1995;40:1707–15.
- de Pouvourville G, Minvielle É. Measuring the quality of hospital care: The state of the art. What information should be made available to the public? Measuring Up: Improving Health System Performance in OECD Countries Paris: Organisation for Economic Co-operation and Development; 2002. 251–75.
- Iezzoni LI. Reinvigorating the quality improvement incentives of hospital prospective payment. Med Care. 2009;47: 269–71.
- Terris DD, Aron DC. Attribution and causality in health-care performance measurement. Smith PC, Mossialos E, Papanicolas I, Leatherman S. Performance Measurement for Health System Improvement: Experiences, Challenges, and Prospects. Cambridge: Cambridge University Press; 2009. 311–38.
- Gissler M, Haukka J. Finnish health and social welfare registers in epidemiological research. Norsk Epidemiologi. 2004;14:113–20.
- Haukka J, Suvisaari J, Tuulio-Henriksson A, Lonnqvist J. High concordance between self-reported medication and official prescription database information. Eur J Clin Pharmacol. 2007;63:1069–74.
- Rikala M, Hartikainen S, Sulkava R, Korhonen MJ. Validity of the Finnish Prescription Register for measuring psychotropic drug exposures among elderly finns: a population-based intervention study. Drugs Aging 2010;27: 337–49.
- Lahti RA, Penttila A. The validity of death certificates: routine validation of death certification and its effects on mortality statistics. Forensic Sci Int. 2001;115:15–32.
- Gissler M, Shelley J. Quality of data on subsequent events in a routine Medical Birth Register. Med Inform Internet Med. 2002;271:33–8.
- Jämsen E, Huotari K, Huhtala H, Nevalainen J, Konttinen YT. Low rate of infected knee replacements in a nationwide series – is it an underestimate? Acta Orthop. 2009;802: 205–12.
- Pajunen P, Koukkunen H, Ketonen M, Jerkkola T, Immonen-Raiha P, Karja-Koskenkari P, . The validity of the Finnish Hospital Discharge Register and Causes of Death Register data on coronary heart disease. Eur J Cardiovasc Prev Rehabil. 2005;12:132–7.
- Tolonen H, Salomaa V, Torppa J, Sivenius J, Immonen-Raiha P, Lehtonen A, . The validation of the Finnish Hospital Discharge Register and Causes of Death Register data on stroke diagnoses. Eur J Cardiovasc Prev Rehabil. 2007;14: 380–5.
- Sund R, Nurmi-Luthje I, Luthje P, Tanninen S, Narinen A, Keskimaki I. Comparing properties of audit data and routinely collected register data in case of performance assessment of hip fracture treatment in Finland. Methods Inf Med. 2007;46:558–66.
- Pihlajamaa J, Suvisaari J, Henriksson M, Heila H, Karjalainen E, Koskela J, . The validity of schizophrenia diagnosis in the Finnish Hospital Discharge Register: findings from a 10-year birth cohort sample. Nord J Psychiatry. 2008;62:198–203.
- Sund R. Methodological perspectives for register-based health system performance assessment – Developing a hip fracture monitoring system in Finland. Helsinki: Stakes, 2008. Research Report. 174.
- Jacobzone S. Introduction to the ageing-related diseases project. A Disease-based Comparison of Health Systems Paris: Organisation for Economic Co-operation and Development; 2003. 11–26.
- Solon JA, Feeney JJ, Jones SH, Rigg RD, Sheps CG. Delineating episodes of medical care. Am J Pub Health. 1967;57:401–8.
- Rosen AK, Mayer-Oakes A. Episodes of care: theoretical frameworks versus current operational realities. Jt Comm J Qual Improv. 1999;25:111–28.
- Sund R. Utilisation of administrative registers using scientific knowledge discovery. Intel Data Analysis. 2003;7: 501–19.
- Rosen AAB, Cutler DM. Challenges in building disease-based national health accounts. Med Care. 2009;47 (supplement):S7–S13.
- Hujanen T. Terveydenhuollon yksikkökustannukset Suomessa vuonna 2001. Stakes, Aiheita 1, 2003 [in Finnish].
- Sund R, Juntunen M, Lüthje P, Huusko T, Häkkinen U. Monitoring the performance of hip fracture treatment in Finland. Ann Med. 2011;43:S39–46.
- Häkkinen U, Hartikainen J, Juntunen M, Peltola M, Tierala I. Analysing current trends in care of acute myocardial infarction using PERFECT data. Ann Med. 2011;43:S14–21.
- de Groot V, Beckerman H, Lankhorst GJ, Bouter LM. How to measure comorbidity: a critical review of available methods. J Clin Epidemiol. 2003;56:221–9.
- Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, . Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43:1130–9.
- Holman CD, Preen DB, Baynham NJ, Finn JC, Semmens JB. A multipurpose comorbidity scoring system performed better than the Charlson index. J Clin Epidemiol. 2005;58:1006–14.
- Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36:8–27.
- Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis. 1987;40: 373–83.
- Maio V, Yuen E, Rabinowitz C, Louis D, Jimbo M, Donatini A, . Using pharmacy data to identify those with chronic conditions in Emilia Romagna, Italy. J Health Serv Res Policy. Policy 2005;10:232–8.
- Lamers LM, van Vliet RC. The Pharmacy-based Cost Group model: validating and adjusting the classification of medications for chronic conditions to the Dutch situation. Health Policy. 2004;68:113–21.
- Naughton C, Bennett K, Feely J. Prevalence of chronic disease in the elderly based on a national pharmacy claims database. Age Ageing. 2006;35:633–6.
- Drösler S, Romano P, Wei L. Health care quality indicators project: Patient safety indicators report 2009. Directorate For Employment, Labour And Social Affairs Health Committee. Paris: OECD, 2009;47.
- DeLong EER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider profiling. Stat Med. 1997;16:2645–64.
- Iezzoni LI. Risk Adjustment for Measuring Healthcare Outcomes. 3rd. Chigago: Health Administration Press; 2003.
- Newhouse JP, McClellan M. Econometrics in outcomes research: the use of instrumental variables. Annu Rev Public Health. 1998;19:17–34.
- Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann. Intern. Med. 1997;127:757–63.
- Brand DA, Quam L, Leatherman S. Medical practice profiling: concepts and caveats. Med Care Res Rev. 1995;52: 223–51.
- Goldstein H, Spiegelhalter DJ. League tables and their limitations: statistical issues in comparisons of institutional performance. J Royal Stat Soc: Series A (Statistics in Society) 1996;159:385–443.
- Christiansen CL, Morris CN. Improving the statistical approach to health care provider profiling. Ann Intern Med. 1997;127:764–8.
- Normand ST, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical care: issues and applications. J Am Stat Assoc. 1997;92:803–14.
- DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider profiling. Stat Med. 1997;16:2645–64.
- Burgess JF Jr, Christiansen CL, Michalak SE, Morris CN. Medical profiling: improving standards and risk adjustments using hierarchical models. J Health Econ. 2000;19: 291–309.
- Ohlssen DI, Sharples LD, Spiegelhalter DJ. A hierarchical modelling framework for identifying unusual performance in health care providers. J Royal Stat Soc: Series A (Statistics in Society) 2007;170:865–90.
- Howley PP, Gibberd R. Using hierarchical models to analyse clinical indicators: a comparison of the gamma-Poisson and beta-binomial models. Int J Qual Health Care. 2003;15:319–29.
- Marshall EC, Spiegelhalter DJ. Institutional performance. Leyland A, Goldstein H. Multilevel Modelling of Health Statistics Chichester: John Wiley & Sons; 2001. 127–42.
- Ash AS, Shwartz M, Peköz EA. Comparing outcomes across providers. Iezzoni LI. Risk adjustment for measuring health care outcomes. 3rd. Chicago: Health Administration Press; 2003. 297–333.
- Keskimäki I, Aro S. Accuracy of data on diagnoses, procedures and accidents in the Finnish Hospital Discharge Register. Int J Health Sci. 1991;2:15–21.
- Juntunen M, Sund R, Peltola M, Häkkinen U. Potilasrakenteen erojen huomioon ottaminen erikoissairaanhoidon vaikuttavuuden rekisteritutkimuksessa. Sosiaalilääk Aikak. 2008;45:258–72.