System impact research – increasing public health and health care system performance

Figures & data

Figure 1. System Impact Research includes all studies assessing performance of the health care or public health systems. All study objects are feasible for Benchmarking Controlled Trials, while many cannot be studied using a Randomized Controlled Trial design. The Clinical Impact Research is placed in the bottom right corner of the figure only to illustrate another category of impact research; i.e. that of assessing impact of interventions targeting individuals.

Figure 2. Shaping the study question of the System Impact Research (SIR) according to PICOS (Population, Index System, Comparator System, Outcome, Study design) -framework. The most important outcome measures are related to six concepts: accessibility, quality of the services (particularly according to scientific evidence), effectiveness (including patient experience), safety, efficiency and equality (of obtaining effective services of uniform quality).

Table 1. Methodological issues in the system impact research (SIR) in benchmarking controlled trials (BCTs) and (cluster) randomized controlled trials (RCTs).

A: Issues related to the research question and study design

1. Benchmarking Controlled Trial or (cluster) Randomized Controlled Trial

2. Operationalized according to PICOS: Population, Index System, Comparator System, Outcome, Study design

B: Issues related to the clinical data:

1. Selection of patients or population to the study and measures to increase comparability

2. Validity and completeness of clinical baseline data, and comparability of study subjects at baseline

3. Validity and completeness of clinical process data throughout the clinical pathway

4. Validity and completeness of clinical outcome data

5. Statistical and data issues

C: Issues related to the features of the health and social care system:

1. Financing of the care system

2. Organization of the care system

3. Available resources

4. Reimbursement and incentives

5. Regulations

6. Competence, evidence-based-medicine, quality improvement, benchmarking (real-effectiveness medicine framework)

7. Other system related issues

Table 2. PICOS (population, index system, comparator system, outcome, and study design) in system impact research. Some examples from benchmarking controlled trials (BCTs) and (cluster) randomized controlled trials (RCTs).

System features/changes	Population	Index system	Comparator system	Outcomes (primary)	Study design
1. Related to the financing of the care system (e.g. tax based or insurance based system)	Hypertensive patients at community health centres (Li et al. Medicine (Baltimore), 2015;94;e455)	Government-funded system	Hospital- or private-funded system	Quality of hypertension management and control of hypertension	Benchmarking controlled trial (BCT)
2. Related to the reimbursement and incentives (e.g. pay for performance)	Population of north-west region of England vs rest of England (Sutton et al. NEJM 2012;367:1821–1828)	‘Advancing quality’ – a hospital pay for performance program	No hospital pay for performance program	Mortality	Benchmarking controlled trial (BCT)
3. Related to the way how and by whom the services are organized/provided (e.g. centralized vs decentralized)	Patients with chronic obstructive pulmonary disease (Kruis et al. BMJ 2014; 349:g5392)	Integrated disease management delivered in primary care	Usual care	Quality of life	Randomized controlled trial (RCT; multicentre, pragmatic cluster randomized controlled trial)
4. Related to the regulations (e.g. on uptake of new technology)	Surgical patients in the NSQIP of database of American College of surgeons (Rajaram et al. JAMA 2014;312:2374–2384)	Restricted resident duty hours at surgical departments: two years after the 2011 duty hour reform	Previous duty hours at surgical departments: two years before the 2011 duty hour reform.	Composite measure of death or serious morbidity within 30 days of surgery	Benchmarking controlled trial (BCT)
5. Related to the amount of available resources for health care (e.g. amount of personnel, GDPs of the countries)	Intensive care patients (Wallace et al. NEJM 2012;366:2093–2101)	Nighttime intensivist physician staffing	No nighttime intensivist physician staffing	Mortality	Benchmarking controlled trial (BCT)
6. Related to competence of the staff, use of up-to-date evidence, quality improvement and benchmarking (real-effectiveness medicine framework)	Bariatric surgery patients (Birkmeyer et al. NEJM 2013;369:1434–1442)	Bariatric surgery performed by surgeons belonging to the top quartile (of all participating surgeons) in their surgical skills	Bariatric surgery performed by surgeons belonging to the bottom quartile (of all participating surgeons) in their surgical skills	Complication rates after bariatric surgery	Benchmarking controlled trial (BCT)
7. Related to other system or structure related features	Hospital personnel; patients (Pittet et al. Lancet 2000; 356:1307–1312)	Hospital before implementation of a hand-hygiene campaign	Hospital during implementation of a hand-hygiene campaign	Compliance with hand hygiene during routine patient care; infection rates among patients	Benchmarking controlled trial (BCT)