Validation of the Oxford Participation and Activities Questionnaire

Abstract

Purpose

There is growing interest in the management of long-term conditions and in keeping people active and participating in the community. Testing the effectiveness of interventions that aim to affect activities and participation can be challenging without a well-developed, valid, and reliable instrument. This study therefore aims to develop a patient-reported outcome measure, the Oxford Participation and Activities Questionnaire (Ox-PAQ), which is theoretically grounded in the World Health Organization’s International Classification of Functioning, Disability, and Health (ICF) and fully compliant with current best practice guidelines.

Methods

Questionnaire items generated from patient interviews and based on the nine chapters of the ICF were administered by postal survey to 386 people with three neurological conditions: motor neuron disease, multiple sclerosis, and Parkinson’s disease. Participants also completed the Medical Outcomes Study (MOS) 36-Item Short Form Health Survey (SF-36) and EQ-5D-5L.

Results

Thus, 334 participants completed the survey, a response rate of 86.5%. Factor analysis techniques identified three Ox-PAQ domains, consisting of 23 items, accounting for 72.8% of variance. Internal reliability for the three domains was high (Cronbach’s α: 0.81–0.96), as was test–retest reliability (intraclass correlation: 0.83–0.92). Concurrent validity was demonstrated through highly significant relationships with relevant domains of the MOS SF-36 and the EQ- 5D-5L. Assessment of known-groups validity identified significant differences in Ox-PAQ scores among the three conditions included in the survey.

Conclusion

Results suggest that the Ox-PAQ is a valid and reliable measure of participation and activity. The measure will now be validated in a range of further conditions, and additional properties, such as responsiveness, will also be assessed in the next phase of the instrument’s development.

Keywords:

• activity
• participation
• PROM
• patient-reported outcome measure
• questionnaire
• FDA
• ICF
• validity
• reliability

Introduction

The Oxford Participation and Activities Questionnaire (Ox-PAQ) is a newly developed patient-reported outcome measure, theoretically grounded in the World Health Organization’s International Classification of Functioning, Disability, and Health (ICF).¹ It is intended for generic use with patients experiencing a broad range of health conditions. The background and rationale for the measure have previously been published in a study protocol,² which readers may wish to refer to. In brief, however, current measures of participation and activity lack theoretical underpinning and are largely disability and rehabilitation focused.^3–8 Additionally, there is no measure of participation and activity for generic use, which fully meets current standards set by regulatory bodies such as the US Food and Drug Administration⁹ and the European Medicines Agency.¹⁰

The item generation process and pretesting procedures for the Ox-PAQ have been extensively reported elsewhere.^11–13 In summary, semistructured interviews were conducted with 37 people experiencing a range of conditions, including arthritis, cancer, chronic back pain, diabetes, motor neuron disease (MND), multiple sclerosis (MS), Parkinson’s disease (PD), and spinal cord injury. These interviews generated a preliminary pool of 222 items, which was subsequently reduced to 24 items via an iterative process in meetings between the authors. The resulting items were pretested through an expert review panel, a translatability assessment, and a series of 13 cognitive interviews. The pretesting procedures led to minor changes to a number of items and the addition of four new questions, resulting in a draft measure of 28 items, answerable on a five-point Likert scale, for validation in a large-scale survey.

The aim of this study is to make the first psychometric assessment of the Ox-PAQ through its administration to people with one of three neurological conditions: MND, MS, and PD. MND is a chronic degenerative neurological condition characterized by progressive degeneration of the upper and lower motor neurons in the brain and spinal cord, resulting in rapid and severe disability. The majority of people with MND die of respiratory muscle weakness <3 years from the onset of symptoms.¹⁴ MS is a chronic condition generally characterized by recurrent relapses followed by remissions, although ∼20% of patients experience a chronic progressive form. People with MS (PwMS) can experience both physical and emotional symptoms, including chronic fatigue and depression, with a significant proportion requiring assistance with walking within 15 years of onset.^15–17 PD is a chronic progressive condition characterized by tremor, bradykinesia, and rigidity. People with PD (PwP) are susceptible to psychiatric symptoms such as depression, hallucinations, and confusion, as well as the likelihood of falls and freezing of gait as the condition progresses.^18,19 Considering the clinical characteristics of the conditions outlined, all three clearly have the potential to have a significant impact on participation and activity in a number of distinct ways, rendering them ideal candidates with which to test the Ox-PAQ.

The specific aims of this study are threefold. First, we aim to identify the underlying factor structure of the Ox-PAQ through the use of factor analysis techniques. Second, we aim to make an assessment of both the internal and external reliability levels of the new measure. Finally, we test the validity of the Ox-PAQ by assessing the magnitude of association with other similarly related constructs alongside an assessment of groups hypothesized to differ; specifically, considering the disparate nature of the disease groups outlined earlier (MND, MS, and PD), it is hypothesized that there will be significant differences in the Ox-PAQ scores between the three conditions.

Methods

Ethical approval for this stage of the Ox-PAQ study was granted by the Medical Sciences Inter Divisional Research Ethics Committee of the University of Oxford (reference MSD-IDREC-C1-2014-089).

Participants

Recruitment of participants was undertaken over a period of 6 months with the assistance of three patient support organizations: the Motor Neuron Disease Association, MS Society, and Parkinson’s UK. The organizations advertised the study through various means, including social media, Web sites, print and electronic publications, research bulletin boards, and emails, inviting potential participants to contact the research team to express their interest in taking part.

Inclusion/exclusion criteria

Participants were required to have a confirmed diagnosis of MND, MS, or PD, as well as the ability to complete the survey independently. Participants were also required to be competent in the use of English, be aged ≥18 years, and be living in the UK.

Materials

A survey booklet consisting of four sections was administered; demographic data (sex, age, age at diagnosis, marital status, and ethnic origin), the Ox-PAQ (as detailed earlier), and two further instruments for the purpose of evaluating its validity.

MOS 36-Item Short Form Health Survey

The MOS SF-36^20,21 is a 36-item questionnaire comprising eight domains of health: Physical Functioning, Role Physical, Role Emotional, Social Functioning, Mental Health, Energy/Vitality, Pain, and General Health Perception. Response options vary across items, from a simple dichotomous yes/no response to a six-point Likert scale. Raw scores for each health domain are transformed to obtain a range from zero to 100, with higher scores indicating superior health status. The measure has been widely adopted in numerous research studies and demonstrates excellent psychometric properties.²²

EQ-5D-5L

EQ-5D-5L^23,24 is a five-item generic measure assessing mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Initially developed with questions answered on a three-point Likert scale, a revised version of the measure now incorporates a five-point Likert scale. The EQ-5D-5L includes a visual analog scale to indicate general health, with a score of zero reflecting worst health status and 100 indicating the best possible health status. Recent studies^25–29 suggest that the updated measure is both valid and reliable.

Procedure

After contacting the research team by telephone or email, participants were sent the booklet of questionnaires and a written consent form for completion and return. A follow-up email or letter was sent to nonresponders after 2 weeks. Participants who agreed to take part in a test–retest procedure were sent the Ox-PAQ again 2 weeks after receipt of their original questionnaire booklet.

Statistical analysis

Data were checked for normality of distribution and presence of outliers prior to statistical analysis. Missing values, as well as floor and ceiling effects, were calculated for each item of the Ox-PAQ. Raw scores were transformed to a range from zero to 100, with higher scores indicative of inferior functioning. Principal components analysis (PCA) with varimax rotation was performed to identify the underlying construct of the measure. The internal reliability of identified domains was assessed via corrected item–total correlations (ITCs) and Cronbach’s alpha.³⁰ Test–retest reliability was calculated using the single-measures (two-way mixed-effects model) intraclass correlation coefficient (ICC).³¹ Concurrent validity was determined through calculation of Pearson correlations³² between the Ox-PAQ and the two instruments MOS SF-36^20,21 and EQ-5D-5L.^23,24 Known-groups validity was assessed through calculation of one-way analysis of variance (ANOVA) and Tukey’s post hoc tests. Data were analyzed using SPSS Version 20 (IBM Corporation, Armonk, NY, USA).³³

Results

A total of 334 participants completed the postal survey, with a response rate of 86.5%. Mean age was 60.06 years (standard deviation [SD]: 12.10 years; range: 24–88 years), mean age at diagnosis was 52.82 years (SD: 14.50 years; range: 18–87 years), and mean disease duration was 7.31 years (SD: 7.52 years; range: 0–50 years). The sample comprised 162 males (48.5%) and 172 females (51.5%). Further sample characteristics by disease group can be viewed in Table 1.

Table 1 Sample characteristics by disease group

Condition	N	Male: female	Mean age, yearsa	Mean age at diagnosis, yearsa	Mean disease duration, yearsa
MND	97	65:32	62.96 (8.93; 42–80)	60.18 (9.66; 35–78)	2.78 (4.11; 0–27)
MS	100	21:79	49.12 (11.26; 24–80)	36.98 (9.71; 18–63)	11.94 (9.32; 0–50)
PD	137	76:61	66.12 (8.83; 40–88)	59.60 (10.35; 30–87)	6.93 (5.57; 0–31)
Total sample	334	162:172	60.06 (12.10; 24–88)	52.82 (14.50; 18–87)	7.31 (7.52; 0–50)

Note:

a The values are represented as mean (standard deviation; range).

Abbreviations: MND, motor neuron disease; MS, multiple sclerosis; PD, Parkinson’s disease.

Percentages of missing responses, as well as floor and ceiling effects, for each of the 28 items of the Ox-PAQ are presented in Table 2. Missing data were minimal, ranging between 0% and 1.8%. Items 2, 21, and 22 (highlighted with asterisk) were subsequently removed from further analysis due to floor effects >40%. A preliminary PCA of the remaining 25 Ox-PAQ items was performed as a means of identifying the underlying construct (scale structure) of the measure. Based on inspection of factors by two of the authors (DM and CJ), two further items, relating to making small movements with hands and coping with pain, were removed due to lack of relevance with the factor onto which they loaded. A further PCA of the remaining 23 Ox-PAQ items resulted in a three-factor solution, explaining 72.7% of variance. Item factor loadings and percentage of explained variance by factor can be viewed in Table 3. Factor 1, Routine Activities (14 items), assesses individuals’ capacity to engage in regular activities that form the basis of daily life. Factor 2, Emotional Well-Being (five items), provides a snapshot of current mental health status. Factor 3, Social Engagement (four items), reflects how well, or otherwise, individuals are able to maintain relationships, both personal and from a wider community perspective.

Table 2 Percentage of missing data and floor/ceiling effects by Ox-PAQ item

Item no	% Missing responses	% Floor	% Ceiling
1	0.3	28.2	16.5
2	0.0	46.7a	14.1
3	0.0	33.2	17.4
4	0.0	34.4	16.5
5	0.3	22.2	30.0
6	0.3	19.5	16.8
7	0.6	29.5	26.8
8	0.6	14.5	26.5
9	1.2	24.5	31.8
10	0.3	25.8	14.7
11	0.0	25.4	10.2
12	0.6	13.3	38.6
13	0.0	38.0	8.4
14	0.3	39.9	7.2
15	0.9	34.4	11.2
16	0.0	15.3	32.3
17	0.9	27.8	21.8
18	1.2	32.7	22.7
19	1.8	35.7	28.4
20	0.3	36.9	9.3
21	0.3	40.2a	7.8
22	0.0	42.5a	6.6
23	0.9	22.3	7.3
24	0.0	16.8	18.6
25	0.0	13.2	6.3
26	0.3	15.6	6.6
27	0.0	13.8	6.9
28	0.0	28.1	6.0

Note:

a Item removed.

Abbreviation: Ox-PAQ, Oxford Participation and Activities Questionnaire.

Table 3 PCA solution, factor loadings, and percentage of explained variance for the Ox-PAQ

Domain	Item	Factor loading	% Explained variance	% Cumulative variance
Routine activities			57.17	57.17
Doing household chores		0.887
Going to shops		0.873
Physical activities for		0.841
enjoyment
Daily activities you like to do		0.834
Getting around home		0.813
Being as independent as		0.781
would like
Getting dressed		0.755
Doing work, paid or unpaid		0.746
Using public transport		0.736
Engaging in community life		0.642
Using own transport		0.632
Social life		0.607
Leisure activities		0.565
Getting up in the morning		0.558
Emotional well-being			9.64	66.82
Anxious		0.865
Sad		0.861
Depressed		0.856
Stressed		0.829
Control over life		0.598
Social engagement			5.95	72.77
Communicating with others		0.852
Engaging in the community		0.791
Maintaining friendships		0.631
Maintaining close relationships		0.566

Abbreviations: Ox-PAQ, Oxford Participation and Activities Questionnaire; PCA, principal components analysis.

Reliability

Internal reliability

Corrected ITCs and Cronbach’s alpha values for each domain can be viewed in Table 4. ITCs ranged from 0.87 to 0.60, with Cronbach’s alpha values for the three identified domains ranging from 0.81 to 0.96.

Table 4 Ox-PAQ item–total correlations, Cronbach’s alpha values, and domain mean scores

Draft Ox-PAQ no	Item	Corrected ITC	α	Mean score (SD)
Routine activities (14 items)			0.96	47.44 (29.95)
5	Doing household chores	0.87
7	Going to shops	0.86
12	Physical activities for enjoyment	0.82
8	Daily activities you like to do	0.86
4	Getting around home	0.80
16	Being as independent as would like	0.85
3	Getting dressed	0.78
9	Doing work, paid or unpaid	0.78
19	Using public transport	0.80
17	Engaging in community life	0.60
18	Using own transport	0.73
10	Social life	0.80
11	Leisure activities	0.77
1	Getting up in the morning	0.66
Emotional well-being (five items)			0.92	44.02 (25.59)
26	Anxious	0.83
27	Sad	0.84
28	Depressed	0.80
25	Stressed	0.85
24	Control over life	0.71
Social engagement (four items)			0.81	35.26 (26.74)
20	Communicating with others	0.53
15	Engaging in the community	0.64
14	Maintaining friendships	0.72
13	Maintaining close relationships	0.67

Abbreviations: ITC, item–total correlation; Ox-PAQ, Oxford Participation and Activities Questionnaire; SD, standard deviation.

External reliability

Test–retest reliability was assessed in 127 participants who indicated no change in health status when completing the Ox-PAQ 2 weeks after their first completion. ICCs were calculated at 0.96 for Routine Activities, 0.83 for Emotional Well-Being, and 0.83 for Social Engagement.

Validity

Concurrent validity

Pearson correlations between the Ox-PAQ and MOS SF-36 are presented in Table 5. Correlations ranged from –0.41 to –0.87, all being highly statistically significant. Domains of the MOS SF-36 deemed most similar to those of the Ox-PAQ correlated more highly, eg, Physical Function and Routine Activities (r=–0.87, P<0.001), Emotional Well-Being and Emotional Well-Being (r=–0.81, P<0.001) and Social Function and Social Engagement (r=–0.71, P<0.001).

Table 5 Pearson correlations between domains of the Ox-PAQ and MOS SF-36

Domain	Physical function	Role limitation, physical	Role limitation, emotional	Energy/fatigue	Emotional well-being	Social function	Pain	General health
RA	–0.87	–0.59	–0.41	–0.62	–0.48	–0.76	–0.49	–0.58
EWB	–0.47	–0.49	–0.62	–0.67	–0.81	–0.59	–0.47	–0.58
SE	–0.61	–0.49	–0.49	–0.52	–0.51	–0.71	–0.48	–0.53

Note: All correlations significant at P<0.001.

Abbreviations: EWB, Emotional Well-Being; MOS SF-36, Medical Outcomes Study 36-Item Short Form Health Survey; Ox-PAQ, Oxford Participation and Activities Questionnaire; RA, Routine Activities; SE, Social Engagement.

Pearson correlations between the Ox-PAQ and EQ-5D-5L are presented in Table 6. Correlations range from 0.43 to 0.81, all being highly statistically significant. As with the MOS SF-36, those EQ-5D-5L items deemed most similar to those of the Ox-PAQ correlated more highly, eg, Mobility and Routine Activities (r=0.81, P<0.001), Usual Activities and Routine Activities (r=0.79, P<0.001) and Anxiety/Depression and Emotional Well-Being (r=0.75, P<0.001).

Table 6 Pearson correlations between domains of the Ox-PAQ and the EQ-5D-5L

Domain	Mobility	Self- care	Usual activities	Pain/discomfort	Anxiety/depression	VAS
RA	0.81	0.79	0.79	0.46	0.47	0.72
EWB	0.45	0.43	0.49	0.46	0.75	0.60
SE	0.58	0.56	0.59	0.44	0.48	0.57

Note: All correlations significant at P<0.001.

Abbreviations: EWB, Emotional Well-Being; MOS SF-36, Medical Outcomes Study 36-Item Short Form Health Survey; Ox-PAQ, Oxford Participation and Activities Questionnaire; RA, Routine Activities; SE, Social Engagement; VAS, visual analog scale.

Known-groups validity

Mean Ox-PAQ domain scores and standard deviations by disease group are given in Table 7. ANOVA results indicate statistically significant differences among the three conditions for all three domains: Routine Activities: F(2,311)=45.66, P<0.001; Emotional Well-Being: F(2,330)=10.64, P<0.001; Social Engagement: F(2,326)=14.16, P<0.001. Post hoc tests (Tukey’s honest significant difference) at the 0.05 level of significance confirm significantly inferior scores in Routine Activities for people with MND when compared to PwMS (P<0.001) and PwP (P<0.001), alongside significantly inferior scores for PwMS compared to PwP (P<0.001). For Emotional Well-Being, significantly inferior scores are evident when comparing those with MND and PwP (P<0.001), as well as PwMS and PwP(P<0.001). Assessment of Social Engagement identifies significantly inferior scores for people with MND compared to PwMS (P<0.001) and PwP (P<0.001).

Table 7 Mean Ox-PAQ scores and standard deviations by disease group

Ox-PAQ Domain	Condition	Mean	Standard deviation
Routine activities	MND	67.43	27.75
	MS	49.66	27.38
	PD	32.80	24.78
Emotional well-being	MND	51.13	25.15
	MS	47.10	26.71
	PD	36.69	23.22
Social engagement	MND	46.87	28.64
	MS	33.02	24.84
	PD	28.86	24.19

Abbreviations: MND, motor neuron disease; MS, multiple sclerosis; Ox-PAQ, Oxford Participation and Activities Questionnaire; PD, Parkinson’s disease.

Discussion

This study has presented the first psychometric evaluation of the newly developed Ox-PAQ. Before identifying the underlying factor structure of the new measure, the percentages of missing responses, as well as the floor and ceiling effects, for the original 28 items were inspected. Percentage of missing data by item was low, with no item exceeding 2%, indicating a high level of acceptability to respondents. Analysis of floor and ceiling effects led to the removal of three items, due to floor effects exceeding 40%, a criterion incorporated in the validation of previous measures.^34,35 Following a preliminary PCA, two further items were removed due to a lack of relevance with the factor onto which they loaded. Twenty-three items were subsequently included in a further PCA to confirm the factor structure of the Ox-PAQ, resulting in a three-factor solution. All factor loadings were in excess of the 0.55 level regarded as good, with the majority higher than the 0.71 level regarded as excellent.³⁶

Reliability of the Ox-PAQ is demonstrated through a number of analyses. The internal reliability of the measure is confirmed through ITCs, which are in excess of previously defined criteria,³⁷ confirming that item scores within each domain are related to the overall domain score. Further evidence is provided by the Cronbach’s alpha values, which lie between 0.81 and 0.96 for the three Ox-PAQ domains, indicating good-to-excellent internal reliability.³⁸ ICCs that fall between 0.83 and 0.92 for the three Ox-PAQ domains indicate excellent external reliability and are significantly greater than the recommended level of 0.60.³⁹

Validity of the Ox-PAQ is demonstrated via assessment of concurrent and known-groups validity. Correlations with the MOS SF-36 and EQ-5D-5L indicate strong concurrent validity. The majority of correlations between Ox-PAQ domains and those of the MOS SF-36 and EQ-5D-5L fall in the 0.40–0.60 range typically observed, with the most similarly related domains in excess of the 0.60 level, representing a high degree of concurrent validity.⁴⁰ Assessment of known-groups validity is made where there are good reasons to hypothesize that scores on a measure of interest will differ between groups,⁴¹ as has been incorporated in previous research.^42–44 Previous studies have made comparisons between PwMS and PwP,^45,46 with results reported here largely confirming this previous research; MS can have a significantly greater impact on physical functioning and emotional well-being than PD. Although no study appears to have compared MND with other neurological conditions, considering its clinical characteristics (as outlined in the “Introduction” section), it would seem reasonable to hypoth-esize that scores are likely to be significantly inferior to the scores of PwMS and PwP. Results from the study would seem to confirm this, with people with MND reporting significantly greater problems as measured by all three domains of the Ox-PAQ when compared to PwMS and PwP.

A number of limitations of this study are acknowledged. First, the reported analyses are confined to three neurological conditions, namely, MND, MS, and PD. Further assessment and validation in alternative disease groups is required to facilitate wider use of the new measure. Additionally, current analyses are confined to traditional psychometric techniques. Further investigation into the operating characteristics of the Ox-PAQ using modern techniques such as Rasch analysis^47–49 may be beneficial in due course. Finally, it is recognized that the method of recruitment for the study was self-selecting in nature, and the sample may not therefore be fully representative of the disease groups that participated.

Conclusion

In conclusion, results from this first psychometric analysis of the Ox-PAQ are promising, with results indicating that the instrument is a valid and reliable measure of participation and activity. The next phase of the instrument’s development will involve migration of the Ox-PAQ to an e-based format, alongside validation in a range of further conditions and an assessment of the responsiveness of the measure. Further details regarding the development and validation of the Ox-PAQ can be found at the University of Oxford Health Services Research Unit Web site http://www.ndph.ox.ac.uk/research/health-services-research-unit-hsru/research/oxpaq-initiative. Information regarding the use of the Ox-PAQ can be obtained from the authors DM or CJ.

Acknowledgments

Development and validation of the Ox-PAQ were funded by the European Brain Council. We acknowledge the following organizations for their continued support throughout the Ox-PAQ study: Floura Health Care, Macmillan Cancer, MND Association, MS Society, Spinal Injuries Association, and Parkinson’s UK. We would also like to acknowledge the continued support and assistance of Dr Mary Baker MBE, Immediate Past President, European Brain Council. Finally, we wish to thank the hundreds of participants who so readily gave their time to take part in the study.

Disclosure

The authors report no conflicts of interest in this work.