PERFECT – Conclusions and future developments

The PERFECT project

The PERFECT project indicates the usefulness of a bottom-up approach to performance measurement, especially when the main interest is in the measurement of the process, cost and outcomes of health care episodes of specific diseases. A good measurement of costs and outcomes requires follow-up of patients through the whole episode, which can be done only when register data from different sources covering the whole population can be linked. Compared, for example, with other national and international quality databases – such as those of the OECD Health Care Quality Indicator Project (HCQI) and Agency For Healthcare Resources and Quality (AHRQ) – the PERFECT project has gone much deeper into the selected diseases, and also includes both long-term outcomes and costs. Thus PERFECT project makes feasible to relate the cost and outcomes.

The performance data can be used for informing health policy in at least two ways. Firstly, it can be used in best-practice benchmarking, in which the various aspects of organisations’ processes will be evaluated in relation to the best practice in a particular disease or health problem. Secondly, it can be used to investigate which measurable and policy-driven factors explain the differences between the hospitals and regions. The two approaches complement each other and demonstrate that the indicators are well-suited to the objectives of health policy.

The PERFECT project has given a new dimension to the benchmarking of care: the data directly help the local decision-makers, since they can compare their own performance by using not only cost or process indicators, but also outcomes and information on the relationship between costs, processes and outcomes. For managerial purposes, the disease-based approach will have great potential and when it is linked with the bottom-up approach, it gives an incentive for clinicians to collect high-quality data (1).

The experience from AMI and stroke patients indicates that about 20–30% of costs could be contained if all regions (hospital districts) would have the same cost as that of the cheapest region – in material where the baseline risk of patients has been adjusted. Similarly, a total of some 500 deaths (amounting to about 7,000 additional life years) would have been avoided if all regions would have the same outcome as the best region in the treatment of the two disease groups in Finland, a country with a comparatively small population (about five million) (2). Since the regional differences in cost did not clearly correlate with regional differences in outcome (mortality), these figures point to great potential for efficiency improvements, i.e. cost can be contained without compromising success in outcomes, and furthermore better outcomes for patients can be obtained without increasing the costs. In addition, it has been found that the centralisation of care in the Neonatal Intensive Care Units in the five university hospitals will decrease the one-year mortality of infants (3). The practical consequences of the project are illustrated in an implementation of an auditing process in one university hospital that showed relatively high mortality among low birth weight infants compared with the other university hospitals. Also, there is a clear indication for better outcomes among stroke patients that have been treated at comprehensive stroke centres (4). For knee and hip replacement surgery, annually over 20 per cent of hospital days can be saved if all the hospital districts would decrease their average length of stay during the first hospital episode to the same level as in the hospital district having the shortest stay. For hip fractures, it would be beneficial to centralise rehabilitation to units with more than 25 patients per year (5).

However, research on the reasons for differences in performance between regions and hospitals has thus far been rather modest. Health economic research applying individual patient-level data to the analysis of costs and outcomes at hospital level is currently taking its first steps (6–8). The priority in future research is to explore the extent to which the observable differences between areas, hospitals and years in cost and outcomes are due to health policy-driven factors (introduction of new technologies or medicines, waiting times, etc.) and to what extent they originate from organisational structures and general economic incentives. When the factors influencing differences in costs and outcomes are identified, the effects of different policy actions can be evaluated.

Evidence-based medicine (EBM) has had an enormous impact in encouraging high-quality original research and systematic reviews (9). It has also brought scientific evidence to the core of clinical decision making (10,11). However, EBM is clearly not enough to enhance the effectiveness and cost-effectiveness of health care interventions in the most expeditious way. Using the approach developed in the PERFECT project, it is possible to obtain reliable information on what actually (in the complex world) happens in the treatment of individual patients and how the patients fare after the treatment episode and beyond. It will be also possible to know to what degree the recommendations of clinical guidelines are actually followed by the health provider, and at health care region and national levels. At the same time, in some instances it may be even possible to test how well following the recommendations of clinical guidelines actually leads to better outcomes.

The monitoring systems developed in PERFECT subprojects have certain limitations that are common to all observational register-based studies. Although the available data are of good quality, the registers only contain a limited amount of detailed clinical or provider-specific data that would be useful from the perspective of the performance assessment. Tailored audit data could be used to complement the register-based data. The challenge is to incorporate extra data collection tasks into routine clinical use, and to make sure that the audit data can be linked with the register-based data. One possibility would be to record extra data in a structured form in the electronic medical record, and then collect the data in a centralised register by using an extra sheet in the Hospital Discharge Register.

Another limitation in the register-based monitoring system is that it is intended only for macro (i.e. time trends in the whole country) and meso-level (regions, hospitals) comparisons. This means that the monitoring system provides only population-level information, and it cannot be used for making individual-level treatment decisions during the actual process of care of one patient. Furthermore the PERFECT data can currently be updated only once a year because of current practices in the maintenance of administrative registers. The challenge is to encourage clinicians and decision makers to use the monitoring systems in their everyday work despite these limitations. For the purposes of continuous client-oriented quality assessment and real-time clinical decision support systems, local information systems that contain and allow access to all relevant data in real-time would be needed. Electronic medical records should be developed so that the required data can be collected in routine clinical practice and used to improve the treatment of patients.

By considering the PERFECT project from the perspective of register-based data, i.e. from the perspective of secondary data that are originally collected for some other purpose, the achievements of the project are remarkable. The project provides complete individual-level data for several important health problems on treatment processes across organisational boundaries in health and social care among both public and private providers, with Finnish personal identification codes that allow deterministic record linkage within and between data extracted from all nationwide administrative registers – this would not be currently feasible in many other countries. Moreover, the adoption of the treatment episode concept from the outset of the project has directed the development of monitoring systems towards the patient (client) perspective. The aim has not been just to develop indicators per se, but to do so in the terms of (reconstructed) treatment episodes that can be observed from the register-based data. Incorporating the clinical, technical and methodological perspectives and expertise by using multidisciplinary study groups has resulted in definitions that are important from the clinical point of view and maximise use of the most reliable data in the registers. This has required a great deal of work, but as the articles in this special issue demonstrate, many of the subprojects on different health problems have been successful in providing results that are scientifically valid and, above all, useful in improving treatment practices and outcomes of care.

From the statistical perspective, the main contributions of the project lie in the issues of measurement and operationalisation of secondary data into a form that can be used for performance assessment purposes. As described elsewhere (12,13), the treatment processes observed from the data can be considered as sample paths of the marked point processes. That leads to very general modelling possibilities. For practical reasons, the basic reporting in the PERFECT project has relied on rather straightforward techniques. It is not difficult to suggest several improvements to the selected approach, but it has turned out to be challenging to conduct such improvements in practice for hundreds of indicators in large datasets. For example, the use of hierarchical models that allow shrinkage of risk-adjusted results has been tested, but due to convergence problems in some cases this approach has turned out to require too much manual work to be feasible in routine use. For the selected techniques, the face validity of the results has been found to be very good and sufficient for the purpose.

More advanced statistical approaches have been developed and applied in research projects using PERFECT data. These include the modelling of multivariate responses (13), hierarchical modelling (5,14,15), neural networks (16,17), sequence methods (13,18), approaches based on minimum description length (19), propensity scores (20), instrumental variables (15) and numerous other techniques (21–26). In fact, the PERFECT data offers an excellent sandbox for the development of statistical approaches for analysing register-based data. The value of the important methodological contributions in the PERFECT project will become obvious when similar data are available in other countries.

In conclusion, regardless of the abovementioned limitations, it is obvious that the described approach offers excellent tools for performance assessment and quality improvement. A new international project (EuroHOPE) will apply the approach used in the PERFECT project in international comparative performance analysis.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.