Co-calibrating quality-of-life scores from three pulmonary disorders: implications for comparative-effectiveness research

Abstract

Background Efficient use of health resources requires accurate outcome assessment. Disease-specific patient-reported outcome (PRO) measures are designed to be highly relevant to patients with a specific disease. They have advantages over generic PROs that lack relevance to patient groups and miss crucial impacts of illness. It is thought that disease-specific measurement cannot be used in comparative effectiveness research (CER). The present study provides further evidence of the value of disease-specific measures in making valid comparisons across diseases.

Methods The Asthma Life Impact Scale (ALIS, 22 items), Living with Chronic Obstructive Pulmonary Disease (LCOPD, 22 items) scale, and Cambridge Pulmonary Hypertension Outcome Review (CAMPHOR, 25 items) were completed by 140, 162, and 91 patients, respectively. The three samples were analyzed for fit to the Rasch model, then combined into a scale consisting of 58 unique items and re-analyzed. Raw scores on the three measures were co-calibrated and a transformation table produced.

Results The scales fit the Rasch model individually (ALIS Chi² probability value (p-Chi²) = 0.05; LCOPD p-Chi²=0.38; CAMPHOR p-Chi²=0.92). The combined data also fit the Rasch model (p-Chi²=0.22). There was no differential item functioning related to age, gender, or disease. The co-calibrated scales successfully distinguished between perceived severity groups (p < 0.001).

Limitations The samples were drawn from different sources. For scales to be co-calibrated using a common item design, they must be based on the same theoretical construct, be unidimensional, and have overlapping items.

Conclusions The results showed that it is possible to co-calibrate scores from disease-specific PRO measures. This will permit more accurate and sensitive outcome measurement to be incorporated into CER. The co-calibration of needs-based disease-specific measures allows the calculation of γ scores that can be used to compare directly the impact of any type of interventions on any diseases included in the co-calibration.

Keywords:

• Asthma
• Co-calibration
• Chronic obstructive pulmonary disease
• Comparative effectiveness research
• Item response theory
• Pulmonary hypertension
• Rasch analysis
• γ scores

Introduction

As healthcare resources become more strained by increasing longevity and rates of chronic disease, the ability to evaluate health outcomes accurately becomes more crucial. One approach to the problem is comparative effectiveness research (CER). The central question of CER is which treatment works best, for whom, and in which circumstances¹. While such research is generally limited to alternative interventions for a single disease, CER would be equally valuable where decisions need to be taken about whether to treat Disease A or Disease B in terms of health gains. In the UK, the National Institute of Clinical Excellence (NICE) recommend the use of generic patient-reported outcomes (PRO)—particularly the EQ-5D²—for such research³. However, the psychometric limitations of this PRO mean that it proves difficult to show differences between the effectiveness of alternative interventions^4–6. Recently, NICE reversed its position and postponed implementation of value-based assessment⁷. Similarly, in the US, the Patient Protection and Affordable Care Act specifically ruled out employing Quality Adjusted Life Years using the EQ-5D. Despite this reversal, CER remains crucial to comparing the outcome of different packages of interventions. NICE has now recognized that, for CER to be a worthwhile alternative, sensitive and valid PRO measures are required that take into account the wider impact of disease⁷.

Available generic PROs generally have poor psychometric properties, including problems with response functioning, item functioning, and measurement range^8–10. When generic PROs are used to combine patients with different diseases, items in the scales have exhibited disease-related differential item functioning (DIF)^11–13. The DIF arises because the same question may well be interpreted differently by patients with different diseases. Furthermore, generic measures have relatively poor sensitivity, making it questionable whether generic PROs are suitable for use in CER studies.

Modern PRO indicators focus on disease-specific outcomes¹⁴. Because the indicators can ask highly relevant questions and avoid irrelevant issues they are more valid and reliable than generic outcome descriptors. Evidence also demonstrates that such disease-specific measures are more sensitive to change than generic indicators^15–20. As disease-specific measures are relevant to only one disease it would be expected that they could not be used in CER. The purpose of this study was to assess whether PRO measures developed using Rasch analysis can overcome this shortcoming.

Most PROs were developed using classical test theory (CTT), also known as true score theory²¹. CTT assumes that the observed score for each individual is comprised of their true ability and a random error component²². CTT has been widely applied in PRO development, possibly due to its weak underlying assumptions that require little mathematical knowledge, making it simple to use²³. However, there are several limitations of CTT. The method relies on correlational analyses such as internal consistency and factor analysis. Furthermore, the application of CTT in PRO development produces numbers that constitute only ordinal level data. As a result, the requirements for parametric assessment are not met. Non-parametric analyses are less powerful than parametric analyses and do not justify the calculation of change scores in clinical trials²⁴.

In contrast, Rasch analysis²⁵ gives greater emphasis to item-level information and provides superior detail on the plausible measurement properties of scales^26–28. The Rasch model has particularly strong mathematical characteristics compared with other item response theory (IRT) approaches. When data fit the Rasch model fundamental measurement is achieved and ordinal scores are transformed into interval-level variables^24,29. Rasch analysis also allows responses from different disease-specific PRO measures to be co-calibrated onto the same measurement scale, allowing the impact of different diseases to be combined and compared. Such co-calibration has been widely used in educational settings to equate tests of varying difficulty levels³⁰, but is increasingly also being applied in health research^31–33.

The aim of this study was to determine whether it is possible to co-calibrate scores on three previously validated, disease-specific, needs-based PROs; the Asthma Life Impact Scale (ALIS)³⁴, the Living with Chronic Obstructive Pulmonary Disease (LCOPD) scale³⁵, and the Cambridge Pulmonary Hypertension Outcome Review (CAMPHOR)³⁶ to allow scores from patients with different respiratory conditions to be combined and compared validly. The secondary aim was to assess whether the PROs could be used in respiratory CER studies.

The study used a common item design to co-calibrate the three scales in which respondents only complete one scale, but the scales share common item content. This design has been employed successfully previously in dermatology³¹. In order to co-calibrate scores from different scales using a common item design, the scales must meet strict criteria. They must be based on the same theoretical outcome construct, fit the Rasch model (be unidimensional) and have overlapping items. The three scales used in the study satisfied these requirements³⁷.

Patients and methods

Participants

The asthma (n = 140) and chronic obstructive pulmonary disease (COPD) samples (n = 162) were recruited from Peterborough District Hospital and patient support groups^34,35. The pulmonary hypertension (PH) sample (n = 91) was recruited from Papworth Hospital, Cambridge³⁶. Ethics committee approval was obtained for each sample separately.

Measures

The CAMPHOR is a suite of three measures, including a quality of life (QoL) scale. For the present study, only the QoL scale was employed. The ALIS and LCOPD are disease-specific outcome measures of QoL. All three scales were developed with the needs-based model of QoL as the theoretical basis. The model theorizes that QoL is determined by the ability and capacity of an individual to satisfy particular human needs. QoL is high when these needs are satisfied³⁷.

The ALIS and LCOPD consist of 22 items and the CAMPHOR QoL scale has 25. All response options are dichotomous (‘True’ or ‘Not True’) and overall scores are calculated by summing the individual item scores. High scores on the measures signify poor QoL.

All three scales were developed using the Rasch model.

Procedure

Three stages were involved in the analysis:

1.	Data from the three measures were analyzed individually to confirm their fit to the Rasch model.
2.	Data from the measures were combined onto the same measurement scale using a common item design. Rasch analysis was applied to assess fit statistics for the combined scale.
3.	The validity of the co-calibrated scale was investigated by relating Rasch scores to disease type, perceived general health, and illness duration.

Rasch analysis

The Rasch model is a type of IRT model that can be applied to scales to assess their measurement properties. The response patterns achieved from a set of items are tested against what is expected by the Rasch model. The Rasch model is a probabilistic version of the Guttman scale³⁸. According to the Rasch model, the probability of an individual affirming a particular item is a logistic function of the relative distance between the person location parameter and the item location parameter. In other words, the probability of an individual affirming an item is dependent only on the individual’s level of the construct (e.g., QoL) and the level of the construct assessed by the item. The Rasch model places items in hierarchical order of severity on the same underlying metric. The mathematical unit of the Rasch model is the logit. The mean location of the items on the metric is anchored at ‘0’ and item locations range either side of this fixed point. Within the context of PROs, milder items are represented by negative logit values and more severe items by positive logit values.

Because the three scales had two possible response options for each item, the dichotomous Rasch model was employed. Several analyses were used to assess fit to the Rasch model. Internal reliability was analyzed using the Person Separation Index (PSI). A PSI score of 0.70 is considered the minimum acceptable level³⁹. Mean item and person fit residual statistics provide information on overall fit to the model. The statistics measure the extent to which observed item and person estimates deviate from model expectations. The co-calibrated scale’s overall fit to the model was analyzed and reported as a Chi² statistic. A Chi² probability value (p-Chi²) of less than 0.05 (Bonferroni corrections applied) means that the scale has significantly deviated from the expected pattern and misfits the Rasch model.

Fit statistics for individual items were also investigated. Item fit residuals should lie within ±2.5 if all individuals respond as anticipated. High negative item fit residuals suggest item redundancy, whereas high positive item fit residuals denote multidimensionality. Chi² statistics were employed to assess whether observed scores deviated from model expectations. Probability values below 0.05 (Bonferroni corrections applied) indicate that observed scores for an item deviate from those expected by the Rasch model.

A requirement of the Rasch model is item invariance across groups, which is tested by examining differential item functioning (DIF) for different sample groups. DIF occurs when different groups (e.g., males and females) respond in a different pattern to an individual item, despite having equal levels of the construct being measured (e.g., QoL)²⁶. An analysis of variance (ANOVA) of standardized residuals was used to examine DIF by gender and age group. DIF by disease was investigated for the items common to more than one scale. An ANOVA p-value of below 0.05 (Bonferroni corrections applied) indicates presence of DIF. DIF can be uniform or non-uniform.

Targeting of items to the QoL of respondents was assessed by examining person-item distribution graphs. These show the ordering of both persons and items on the logit scale and indicate whether the items in the scale are well matched to the respondents.

A further test of the unidimensionality of scales is also calculated⁴⁰. In this test, two sub-sets of responses are identified that are highly divergent (based on items loading at either extremes of the first factor of the residual principal components analysis). The two sub-sets of responses are used to create separate estimates for each individual. Paired t-tests are carried out to assess whether the two test sub-sets produced significantly different estimates for each respondent. Where fewer than 5% of the tests are statistically significant the scale is considered unidimensional. A 95% binomial confidence interval is also provided.

The Rasch Unidimensional Measurement Model (RUMM) 2030⁴¹ program was used in the analyses.

Co-calibration of the ALIS, LCOPD, and CAMPHOR

There are 10 common items linking the three scales, all of which were used for the co-calibration. Five of these items are common to the ALIS and LCOPD only, two to the ALIS and CAMPHOR only, and two to the LCOPD and CAMPHOR. One item is common to all three measures, although this is not necessary for co-calibration. As participants only completed one of the measures, their responses are linked via the common or ‘anchor items’. The analyses of fit to the Rasch model (described above) were conducted.

Analysis of the co-calibrated scale

The Rasch location of all items on the combined measurement scale was linearly transformed to allow each item to be expressed as a standardized score that can range from 0–100. This is referred to as the gamma (γ) score below.

Further analyses were conducted to see how the γ scores performed. Patient scores for asthma, COPD, and PH were compared using an ANOVA In addition, γ scores were combined and related to perceived severity (for ALIS and LCOPD) and New York Heart Association (NYHA) Functional Classification (for CAMPHOR), using an ANOVA and t-test respectively. P-values of < 0.01 were used to indicate statistically significant differences between disease groups.

Results

Demographic details of the samples are shown in Table 1. Perceived illness severity was recorded for asthma and COPD samples. For the PH sample the NYHA Functional Classification was used as an indicator of disease severity.

Table 1. Sample characteristics.

	Asthma (n = 140)	COPD (n = 162)	PH (n = 91)
Gender (%)
Male	41 (29.3)	71 (43.8)	27 (29.7)
Female	98 (70.0)	88 (54.3)	64 (70.3)
Missing	1 (0.7)	3 (1.9)	0 (0.0)
Age (years)
Mean (SD)	50.6 (16.2)	69.3 (9.3)	52.6 (15.9)
Range	64.7 (18.6–83.3)	42.8 (48.3–91.1)	63.2 (18.2–81.4)
Disease duration (years)
Mean (SD)	21.3 (15.3)	12.3 (13.7)	7.0 (7.7)
Range	69.8 (1.3 – 71.0)	75.5 (0.5–76.0)	54.0 (0.0–54.0)
Marital status (%)
Married/Living as married	94 (67.1)	98 (60.5)	58 (63.7)
Widowed	11 (7.9)	30 (18.5)	15 (16.5)
Divorced	10 (7.1)	27 (16.7)	6 (6.6)
Single	22 (15.7)	4 (2.5)	11 (12.1)
Missing	3 (2.1)	3 (1.9)	1 (1.1)
Perceived general health (%)
Excellent	17 (12.1)	3 (1.9)	7 (7.7)
Good	49 (35.0)	25 (15.4)	30 (33.0)
Fair	54 (38.6)	77 (47.5)	38 (41.8)
Poor	19 (13.6)	54 (33.3)	16 (17.6)
Missing	1 (0.7)	3 (1.9)	0 (0.0)
Perceived illness severity (%)
Mild	50 (35.7)	16 (9.9)	—
Moderate	54 (38.6)	66 (40.7)	—
Quite severe	26 (18.6)	63 (38.9)	—
Very severe	8 (5.7)	16 (9.9)	—
Missing	2 (1.4)	1 (0.6)	—
NYHA Classification (%)
I/II	—	—	40 (44.4)
III/IV	—	—	50 (55.6)
Missing	—	—	1 (1.1)

Rasch analysis of individual scales

Table 2 shows overall fit statistics for the ALIS, LCOPD and CAMPHOR. All three measures fitted the Rasch model (p-Chi², p > 0. 05) initially. The PSIs were high, indicating that the measures had appropriate levels of reliability.

Table 2. Overall fit statistics.

	Item-trait interaction Chi^2 p-value	No of items	PSI	Item-person fit residuals				Unidimensionality (95% binomial CI)
				Item fit residual		Person fit residual
				Mean	SD	Mean	SD
ALIS	0.05	22	0.89	−0.60	0.93	−0.32	0.82	0.05 (0.01–0.09)
LCOPD	0.38	22	0.88	−0.23	1.08	−0.24	0.73	0.06 (0.03–0.10)
CAMPHOR	0.92	25	0.88	−0.28	0.75	−0.27	0.79	0.10 (0.05–0.14)
Combined sample	0.22	58*	0.90	−0.37	0.95	−0.28	0.79	N/A*
Ideal values	>0.05		>0.85	0.00	1.00	0.00	1.00	<0.05 (Lower CI <0.05)

*Additional test of unidimensionality could not be employed for the combined sample due to the amount of missing data. There are 58 unique items within the three scales.

Item 12 of the LCOPD (‘My illness frightens me’) had a high fit residual (2.9), indicating misfit to the Rasch model. However, the item characteristic curve showed no pronounced deviation between observed and expected scores. After applying Bonferroni corrections, none of the items in any of the three scales showed DIF by age group (below and above median) or gender.

As the fit indicators did not support the removal of items, all items were retained and carried forward to the combined analysis.

Rasch analysis of combined samples

Item and person fit residual statistics for the combined scales are also shown in Table 2. Overall fit to the model was achieved (p-Chi²>0.05) and reliability was good (PSI > 0.85). The LCOPD item (‘My illness frightens me’) continued to have a high fit residual, but was retained as other fit indicators did not support its removal. There was no DIF found for age group (below and above median), gender, or disease.

Targeting of the items to the respondents is displayed in Figure 1. The graph shows that the items had a good spread and were well matched to the person locations on the logit scale. However, there was some lack of coverage at the mild end of the scale. As shown in the figure, the items were well spread across the logit scale, indicating minimal item redundancy. Figure 2 displays the person and item locations for the combined scale. The most severe item in the combined scale was ‘I don’t want to talk to anybody’. The mildest item was ‘I feel dependent on my treatment’. The logit coverage (the distance between the most severe and mildest items) of the combined sample was 7.0.

Figure 1. Targeting of items for the combined scale.

Figure 2. Item map showing the item and person locations for the combined scale.

The logit positions of the common linking items are shown in Table 3. None of the items displayed DIF by gender, age group or disease after Bonferroni corrections were applied. The logit coverage of the common items was 4.4. The most severe common item was ‘My illness affects my close relationships’, whereas the mildest was ‘I have to pace myself’.

Table 3. Logit positions of the common linking items.

Item	Scale	Logit position
My illness affects my close relationships	ALIS & LCOPD	1.23
I feel like I let other people down	ALIS & CAMPHOR	0.20
My illness controls me	ALIS & LCOPD	0.14
My self-confidence is affected	ALIS & LCOPD	0.04
I’m unable to join in activities with  my friends and family	ALIS & CAMPHOR	−0.09
I feel dependent on others	LCOPD & CAMPHOR	−0.49
I can’t do things on the spur of  the moment	LCOPD & CAMPHOR	−0.78
My illness restricts my social activities	ALIS & LCOPD	−1.01
My illness limits the places I can go	ALIS, LCOPD &  CAMPHOR	−1.95
I have to pace myself	ALIS & LCOPD	−3.18

Table 4 shows example transformed γ scores for the three measures. The mean γ scores (standard error) for the individual scales were 43.63 (4.18) for the ALIS, 46.93 (4.12) for the LCOPD, and 54.71 (4.25) for the CAMPHOR. As stated in the Method section, an individual’s score is the number of items they affirmed. A raw score of 10 would equate to a γ score of 42 on ALIS, 45.2 on LCOPD, and 49.4 on CAMPHOR.

Table 4. Example γ scores for the ALIS, LCOPD, and CAMPHOR.

ALIS			LCOPD			CAMPHOR
Raw score	Location	γ score	Raw score	Location	γ score	Raw score	Location	γ score
5	−2.20	29.72	5	−1.81	33.31	5	−1.37	37.35
10	−0.85	42.03	10	−0.50	45.21	10	−0.04	49.37
15	0.34	52.81	15	0.67	55.85	15	1.11	59.83
20	2.00	67.92	20	2.33	70.96	20	2.46	72.14

Analysis of the co-calibrated scale

Figure 3 shows mean γ scores and corresponding 95% confidence intervals based on the standard error of the mean for disease groups, severity groups, and NYHA classifications. High γ scores indicate worse QoL. Comparison of samples showed that γ scores were able to distinguish between disease (F = 30.58, p < 0.001) and disease severity for both asthma (F = 38.16, p < 0.001) and COPD patients (F = 27.58, p < 0.001). PH patients with a NYHA classification of III/IV scored significantly higher than those with a classification of I/II (t = −4.67, p < 0.001). Comparison of samples showed that asthma patients had significantly lower γ scores than patients with COPD (t = −7.38, p < 0.001) and PH (t = −5.07, p < 0.001). However, there were no significant differences in γ scores between COPD and PH patients (t = 1.37, p = 0.17). It should be noted that the samples were not matched for disease severity.

Figure 3. Mean γ scores and 95% confidence intervals by disease, severity group, and NYHA classification.

Discussion

This study successfully co-calibrated three different disease-specific respiratory PRO measures onto the same measurement scale, using Rasch analysis. The combined scale fit the Rasch model and no adjustments to the scale were required. There was no DIF related to age, gender, or disease. The validity of the γ scores generated from the combined scale was assessed by comparing scores by perceived disease severity (for ALIS and LCOPD) and NYHA classification (for CAMPHOR). The scale was able to distinguish between severity groups, providing evidence of its construct validity. The current study supports previous research that has successfully combined two dermatology disease-specific PRO measures³¹.

A common item design was employed to combine the scales due to the overlapping content of the three measures. All three scales were based on the same theoretical construct and developed using the needs-based model of QoL. The needs-based model postulates that QoL is determined by the ability of an individual to satisfy their needs; QoL is high when most needs are fulfilled³⁷. Since its development in 1992, the needs-based model of QoL has been used as the theoretical basis in the development of 20 disease-specific PRO measures. Disease-specific measures include the PRIMUS (multiple sclerosis)⁴², CLIQ (Crohn’s disease)⁴³, RAQoL (rheumatoid arthritis)⁴⁴ and ASQoL (ankylosing spondylitis)⁴⁵. Because needs are universal there are commonly overlapping items in the PRO measures. For example, a loss of autonomy occurs in a number of chronic illnesses. Indeed, overlapping content is a requirement for co-calibration.

A clear application for co-calibrated needs-based measures is in CER. NICE was established in 1999 to produce guidelines on the value (including costs and benefits) of various interventions. The guidelines are used to make decisions about which treatment options to resource⁴⁶. There are many demands on health service resources, but budgets will always be limited. It is, therefore, crucial that decision-makers are able to tell clearly which therapeutic interventions are having the biggest effect. NICE is in the process of considering other outcomes that could be included in value-based assessments and has acknowledged that the wider impact of the disease should be considered⁷. PRO measures arising from the needs-based model of QoL are well suited for this purpose as they assess how a disease affects patients’ lives and limits their ability to meet their needs.

The availability of γ scores would have additional advantages. They would provide a standardized index of outcome that is common across situations. For example, treating asthma with Drug A may result in an average gain of 3 γ. Treating COPD with Drug B may lead to an average improvement of 2 γ. The relative value of exercise programs could be evaluated using the same indices. For example, exercise programs are currently being evaluated for the three respiratory diseases^47–49. Combining disease-specific PRO measures onto the same measurement continuum would allow the effectiveness of such programs on different respiratory conditions to be directly comparable.

Further research is required to establish the validity of such γ scores. However, they could provide the basis for a new outcome technology system. Following co-calibration of the 20 needs-based disease-specific measures, patient scores on any of them could be converted into γ scores. The γ scores would allow direct, valid comparisons across a wide range of diseases. New needs-based instruments could then be added to the system.

Limitations of the study should be noted. The samples were drawn from different sources. In order to co-calibrate scores from different scales using a common item design, the scales must meet strict criteria. They must employ the same theoretical outcome construct, fit the Rasch model (be unidimensional), and have overlapping items. All outcomes scales should have such qualities. However, other than the needs-based measures, few PRO scales fit the Rasch model or have been developed using a consistent theoretical basis. Therefore, the method described in the current study is not applicable with most available PROs.

Conclusion

The impact of different interventions for asthma, COPD and PH can be compared directly in CER using the co-calibration approach demonstrated in this article. Utilizing disease-specific measures for comparing different diseases has advantages over relying on generic PROs in terms of validity and responsiveness.

Transparency

Declaration of funding

No funding was provided for this study.

Declaration of financial/other relationships

The authors are employees of Galen Research Ltd, a research company that develops and evaluates disease-specific patient-reported outcome measures. JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgments

No assistance in the preparation of this article is to be declared.