The Oxford Participation and Activities Questionnaire: study protocol

Abstract

Background

With an ageing population and increasing demands on health and social care services, there is growing importance attached to the management of long-term conditions, including maximizing the cost-effectiveness of treatments. In line with this, there is increasing emphasis on the need to keep people both active and participating in daily life. Consequently, it is essential that well developed and validated instruments that can meaningfully assess levels of participation and activity are widely available. Current measures, however, are largely focused on disability and rehabilitation, and there is no measure of activity or participation for generic use that fully meets the standards set by regulatory bodies such as the US Food and Drug Administration. Here we detail a protocol for the development and validation of a new patient-reported outcome measure (PROM) for assessment of participation and activity in people experiencing a variety of health conditions, ie, the Oxford Participation and Activities Questionnaire (Ox-PAQ). The stages incorporated in its development are entirely in line with current regulations and represent best practice in the development of PROMs.

Methods

Development of the Ox-PAQ is theoretically grounded in the World Health Organization International Classification of Functioning, Disability, and Health. The project incorporates a new strategy of engaging with stakeholders from the outset in an attempt to identify those characteristics of PROMs considered most important to a range of potential users. Items will be generated through interviews with patients from a range of conditions. Pretesting of the instrument will be via cognitive interviews and focus groups. A postal survey will be conducted, with data subject to factor and Rasch analysis in order to identify appropriate dimensions and redundant items. Reliability will be assessed by Cronbach’s alpha and item-total correlations. A second, large-scale postal survey will follow, with the Ox-PAQ being administered in conjunction with generic measures of health status to further test the validity of the measure. The Ox-PAQ will again be administered at 2 weeks to assess test-retest reliability and at 3 months to assess responsiveness.

Conclusion

The development of the Ox-PAQ is a timely one. With increasing emphasis being placed on the importance of keeping people active and participating in daily life, the instrument has the potential for significant uptake. Its primary use is intended to be in clinical trials and for evaluation of interventions targeted at maintaining activity and participation.

Keywords:

• activity
• participation
• patient-reported outcome measure
• questionnaire

Introduction

With an ageing population and increasing demands on health and social care services, growing importance is attached to the management of long-term conditions, including maximizing the cost-effectiveness of treatments. In line with this, there is increasing emphasis on the need to keep people both active and participating in daily life.^1–4 Consequently, it is essential that well developed and validated instruments that can meaningfully assess levels of participation and activity are widely available.

The current literature, however, demonstrates little consensus on how we define “participation” and “activity”. A number of papers have been published with the aim of clarifying or reaching agreement on a clear definition for both terms,^5–7 but with limited success. Some have called for greater clarity between the two concepts,⁸ whilst others have concluded that participation is a complex, elusive, and subjective concept, influenced by one’s individual perspective.⁹ This serves to highlight the importance of investigating the meaning of concepts such as participation and activity with those in whom we are trying to measure it. As previous qualitative research demonstrates, when this is undertaken, the complex nature of what, for example, participation actually means to different individuals becomes evident.¹⁰

One widely used definition of the concepts of participation and activity is to be found in the World Health Organization International Classification of Functioning, Disability and Health (ICF),¹¹ which at least provides a framework within which to construct scales aimed at measuring the two constructs. The ICF defines participation as “involvement in life situations” and activity as “the execution of a task or action by an individual”. Although initially described as two separate concepts, the final version of the ICF merges activity and participation into a single taxonomy and identifies nine domains which are presented in Table 1.

Table 1 Domains of the ICF classification of participation and activity

1.	learning and applying knowledge
2.	General tasks and demands
3.	Communication
4.	Mobility
5.	Self-care
6.	Domestic life
7.	Interpersonal interactions and relationships
8.	Major life areas
9.	Community, social, and civic life

Abbreviation: ICF, International Classification of Functioning, Disability and Health.

There is still significant debate over the ICF model and particularly in relation to whether activity and participation is a single entity, or whether a clear distinction should be made between the two.^12,13 Despite this continuing debate, a number of patient-reported measures of activity and participation have now been constructed, to varying degrees, based on the model.¹⁴ Recently published reviews and clinical commentaries, however, highlight significant limitations in a number of these measures.^15–20 Current measures are largely focused on disability and rehabilitation, with many falling short of the full range of psychometric properties required for patient-reported outcome measures (PROMs) under guidance such as that provided by the US Food and Drug Administration (FDA).²¹ The development of a number of instruments is poorly reported and some have been developed with small sample sizes, which calls into question the legitimacy of the psychometric analyses undertaken. Many have failed to involve patients in generating items, something that is considered fundamental in developing relevant PROMs that accurately reflect the concerns of those who are being assessed.²¹

Such a picture provides a sound rationale for developing a new measure, the Oxford Participation and Activities Questionnaire (Ox-PAQ). The Ox-PAQ initiative aims to address a number of the issues highlighted above by developing a measure theoretically grounded in the ICF for generic use and with patients at the heart of the item-generation process. Additionally, the views of stakeholders will be gained from the outset in an attempt to identify those characteristics of PROMs considered most important to a range of potential users. The measure will subsequently be assessed for the full range of psychometric properties, and sample sizes will allow for detailed and legitimate psychometric analyses. The development process will be entirely in line with FDA guidelines which are now considered best practice, regardless of whether an instrument is being incorporated into, or developed for, trials supporting pharmaceutical label claims.²²

Materials and methods

Ethical approval for the study was granted by the Medical Sciences Inter-Divisional Research Ethics Committee of the University of Oxford (reference MSD-IDREC-C1-2013-064). The study will follow a nine-point development process, as outlined in Table 2. The methodology (participants, materials, procedure, and statistical analysis) as to how this development process will be achieved is detailed below.

Table 2 Developmental stages of the Ox-PAQ initiative

Stage	Activity
1	Stakeholder interviews
2	Patient interviews
3	Item generation and pretesting
4	Pilot-test survey
5	Psychometric analysis of pilot test survey
6	Large-scale field test
7	Psychometric analysis of field test
8	Test-retest assessment
9	Responsiveness assessment

Abbreviation: Ox-PAQ, Oxford Participation and Activities Questionnaire.

Participants

Development of the Ox-PAQ will require three separate groups of participants. The source of recruitment and nature of participants will depend on the particular stage of the project as outlined below.

Stakeholder interviews

A broad range of participants has been drawn from a number of areas where patient-reported outcomes are of relevance (ie, medicine, academia, regulation).

Patient interviews

A sample of patients (approximately 30–40) drawn from a wide range of conditions, including arthritis, cancer, chronic back pain, diabetes, motor neurone disease, multiple sclerosis, Parkinson’s disease, and spinal cord injury, will be recruited via relevant support charities. The patient groups included represent a spectrum of conditions that together affect a number of key bodily systems, as well as having different symptoms, trajectories, and prognoses.

Survey participants

A large separate sample of patients with the conditions outlined above will again be recruited via relevant support charities.

Inclusion/exclusion criteria

Participation in the stakeholder interviews requires participants to have some knowledge and previous experience with PROMs, and be actively engaged in a field where PROMs are of relevance. Inclusion in the patient interview phase will require participants to have a confirmed diagnosis of the condition being investigated. Survey participants will be excluded if they are unable to complete the questionnaires independently and if they do not have English as a primary language. All participants must be aged 18 years or over.

Materials

Two instruments will be administered alongside the Ox-PAQ for the purpose of evaluating its validity.

The SF-36

The SF-36^23,24 is a 36-item instrument containing eight domains (physical functioning, role physical, role emotional, social functioning, mental health, energy/vitality, pain, and general health perception) that has been extensively used and shown to have excellent psychometric properties.^25,26 Raw scores for each domain are transformed to have a range from 0 to 100, with higher scores indicative of superior health status. Two summary scores can also be calculated from the results of the eight domains, ie, the Physical Component Summary and Mental Component Summary.^27,28

The EQ-5d

The EQ-5D²⁹ is a generic, five-item questionnaire that also incorporates a visual analog scale (ranging from 0 [worse health] to 100 [best health]) to indicate general health status. A unique health state is derived by combining responses from each of the five questions. Each score is intended to reflect societal valuations of the health state in terms of “utility” or valuations of health-related quality of life. Any one of the possible 243 health states that can be derived is converted to a summary score, with higher scores indicating superior quality of life. The instrument has acceptable reliability for both the utility (r=0.73) and visual analog scale (r=0.70) components.³⁰

Procedure

The procedure for the nine developmental stages outlined in Table 2 will be conducted as follows.

Semistructured interviews have been conducted across Europe with 18 professionals from a range of different backgrounds and areas of expertise. These have included regulation, commissioning, reimbursement, clinical practice, nursing, health economics, national government, academia, clinical trials, patient support, and health technology assessment. The purpose of these interviews was two-fold: firstly, to discuss issues surrounding current PROMs and, secondly, to consider the broad topic areas a generic measure of participation and activity might include that will be meaningful and considered important. No previous measure has been designed from the outset with the views of such a variety of potential users at its core. Furthermore, the interviews discussed the practicalities of the instrument, such as preferred methods of scoring and rules for interpretation. The interviews will be transcribed, subjected to content analysis, and used to inform later stages of the project.

Exploratory, open-ended, face-to-face, indepth interviews will be undertaken with a sample of patients across a wide range of conditions. The interviews will aim to identify salient dimensions of illness impact that have adversely affected participation and activities. It is anticipated that approximately 30–40 patients will be interviewed in depth until, as recommended, “no new themes emerge”.³¹ Interviews will be recorded, transcribed, and subjected to content analysis. Whilst there will be an interview guide, participants will be free to range across this and address any other relevant topics.

The interviews conducted in Stage 2 will be transcribed and transcripts will be scrutinized independently by the research team for issues relating to illness impact. Issues arising will be recast as questionnaire items to be answered on a five-point Likert scale. A Flesch readability score³² will be calculated for each item with a target of approximately grade 6. A meeting of health care researchers and relevant interested stakeholders will be convened to assess the list of items for completeness, ambiguity, and repetition. The resulting candidate questionnaire will then be discussed with a small group of people representing a range of conditions in a focus group setting for their comments and to assess its face validity. Additionally, the questionnaire will be pretested with a variety of patients in cognitive interviews using the format outlined by Willis.³³ Participants will be asked to complete the questionnaire and explain the reasons for their answers to each question, and comment upon any difficulties or ambiguities they experience.

The long-form questionnaire drawn up in Stage 3 will be mailed to a sample of 600 people with a variety of long-term conditions. This sample size is based on the assumption that the instrument will contain approximately 50 items. Estimates suggest that a minimum of five times as many respondents than items are required for psychometric tests to be meaningful.³⁴ Assuming a typical response rate of between 60% and 70%,³⁵ this will lead to a sample of approximately 360–420 questionnaires, which will permit rigorous testing of the instrument.

Analysis of data from Stage 4 will allow for identification of a shorter form questionnaire using established statistical procedures as detailed below in the Statistical analysis section. Items with high face validity, good correlation with the scale total to which they contribute, and sound reliability will constitute a dimension of the resulting instrument.

The short-form measure generated in Stage 5, together with the SF-36^23,24 and EQ-5D,²⁹ will be mailed to 1,000 patients with a wide variety of conditions. Assuming a 60%–70% response rate (as discussed previously), this will yield a final sample size of approximately 600–700 returned questionnaires.

Further statistical analyses will be undertaken on data from Stage 6 to confirm the results from the initial survey in Stage 5. Minor amendments may be made to the questionnaire at this time.

Previous respondents will be asked to complete the questionnaire once more. If they agree to do so, they will be sent the Ox-PAQ one week after receipt of the measure from Stage 6 in order to assess test-retest reliability. Previous respondents will be asked to complete the questionnaire for the last time after 3 months has elapsed in order to assess the instrument’s responsiveness.

Statistical analysis

Data will be checked for presence of outliers and normality of distribution prior to statistical analysis. Distribution of the data will be analyzed using the Shapiro-Wilk test of normality.³⁶ Subscale and summary scores for the SF-36 and EQ-5D will be calculated according to respective scoring algorithms. Items of the Ox-PAQ will be assessed for floor and ceiling effects under the criteria outlined by Terwee et al.³⁷ Factor analytic techniques will subsequently be performed to assess construct validity and identify dimensions relating to specific aspects of participation and activity. Item-total correlations, corrected for overlap, will be calculated between items and the total score to which they contribute. Rasch analysis^38–40 will also be conducted to determine which items conform to a hierarchical unidimensional structure. Further tests of validity (concurrent and discriminant) will be measured by calculating Pearson’s product-moment⁴¹ or Spearman rank correlation⁴² coefficients (as appropriate depending on data distribution) between the Ox-PAQ, SF-36, and EQ5D. Reliability will be assessed via tests of internal consistency using Cronbach’s alpha coefficient⁴³ and test-retest reliability using intraclass correlations.⁴⁴ Responsiveness to change will be determined through calculation of standardized effect sizes.⁴⁵ Data will be coded in order to maintain confidentiality and analysed using Statistical Package for the Social Sciences version 20 software (IBM, Armonk, NY, USA). Rasch analysis will be conducted using RUMM2030.

Discussion

This paper has described the initial stages of developing and validating the Ox-PAQ, a new measure of participation and activity. The rationale for developing this instrument is relatively straightforward. Current measures are largely focused on disability and rehabilitation, and there is no measure of activity and participation for generic use that fully meets the standards set by regulatory bodies such as the FDA. The Ox-PAQ initiative aims to ameliorate this. The measure will be broadly focused and for use in a range of conditions. The developmental stages discussed follow current guidelines and represent best practice in the development of outcome measures. Additionally, the project incorporates a new strategy of involving and engaging with potential users from the outset of the development process.

The development of the Ox-PAQ is a timely one. With increasing emphasis being placed on the importance of keeping people active and participating in daily life, the instrument has the potential for significant uptake. Its primary use is intended to be in clinical trials and related forms of evaluation of interventions targeted at maintaining activity and participation. Given the increasing importance of electronic data capture, the development of an electronic version of the Ox-PAQ, for use on platforms such as computers, tablets, and smartphones, is something that may follow its initial development and validation. The progress of the development and validation of the Ox-PAQ can be followed at the University of Oxford Health Services Research Unit website (http://www.dph.ox.ac.uk/research/hsru/OxPAQ), and results will be disseminated via international conferences and journals.

Acknowledgments

The Ox-PAQ initiative is funded by the European Brain Council. We would like to acknowledge the continued support and assistance of Dr Mary Baker, President, European Brain Council.

Disclosure

The authors report no conflicts of interest in this work.