Abstract
Objective. To develop and evaluate the Patient Experiences Questionnaire for Out-of-Hours Care (PEQ-OHC) in Norway. Design. Questionnaire development was based on a systematic literature review of existing questionnaires, interviews with users, and expert group consultation. Questionnaire testing followed a postal survey of users who had attended out-of-hours centres in the North, West, and South of Norway. Setting. Primary care out-of-hours services. Subjects. The questionnaire was pre-tested with 13 users and was then mailed to 542 users who had had telephone contact and/or had a consultation with one of three out-of-hours centres. Main outcome measures. Data quality, internal consistency, reliability, and construct validity. Results. The questionnaire was considered to have good content validity by the expert group. There were 225 (41.51%) respondents to the postal questionnaire. Levels of missing data at the item and scale level were acceptable. Principal component analysis supported the four scales of user experiences relating to telephone contact, doctor services, nursing services, and organization. Item-total correlations were all above 0.5 and Cronbach's alpha was above 0.80 for all scales. Statistically significant associations based on explicit hypotheses were evidence for the construct validity of the PEQ-OHC. Conclusion. The development of the PEQ-OHC followed a rigorous process based on a systematic review, interviews with users, and an expert group which lend the questionnaire content validity. The PEQ-OHC has evidence for data quality, internal consistency, reliability, and construct validity.
Questionnaires are increasingly used for assessing patient experiences of out-of-hours care.
Existing questionnaires have limitations relating to data quality, reliability, and validity.
The PEQ-OHC has undergone a rigorous process of development following a systematic review and interviews with patients.
The PEQ-OHC has good evidence for data quality, reliability, and validity.
There have been considerable changes in the organization of out-of-hours primary care in Europe [Citation1–3]. The care setting has moved away from the home to the primary care centre and telephone consultations, which has followed the growth of GP cooperatives and deputizing services [Citation4]. However, the evaluation of developments within out-of-hours care has been limited in scope focusing on the process of care [Citation5], which may stem from the lack of availability of questionnaires that are based on the views of users concerning their experiences or satisfaction with services [Citation4,Citation6,Citation7]. There are several questionnaires that assess user experiences and satisfaction with primary healthcare more generally [Citation8–10] but there is a lack of questionnaires specific to out-of-hours care that have sufficient evidence for data quality, reliability, and validity [Citation7].
According to a systematic review there was just one questionnaire relating to user satisfaction with out-of-hours care that was published before 2005 [Citation7]. The review identified four questionnaires, two from the Netherlands and two from the UK, concluding that all four had limitations in terms of their development and evaluation. The development of the questions within two questionnaires did not include users, which has implications for content validity [Citation8]. Two questionnaires had poor reliability estimates and there was evidence for item redundancy within a third. Evidence for the validity of all four questionnaires was considered to be limited [Citation7].
It is important that these shortcomings are addressed through further development and evaluative work to ensure that such questionnaires meet the necessary criteria for use within evaluative studies and for quality improvement initiatives [Citation11]. Questionnaires that are valid and reliable and based on the views of users [Citation8] are also required for national surveys of user views of out-of-hours care such as that undertaken within the Netherlands [Citation4] and for providers wishing to monitor user experiences and satisfaction as has been recommended in the UK [Citation12].
Norway has a programme of national surveys that measure user experiences of care [Citation13–16]. The survey results are designed to inform patient choice and healthcare quality improvement. The survey questionnaires are based on the views of users and experts and have good evidence for data quality, reliability, and validity [Citation13–16]. A systematic review was undertaken that was designed to inform a survey of users of out-of-hours care in Norway [Citation7]. The work that follows describes the development of the Patient Experiences Questionnaire for Out of Hours Care (PEQ-OHC) and testing for data quality, internal consistency, reliability, and validity. This Norwegian questionnaire is available for forward–backwards translation for countries with a similar provision of out-of-hours care but the relevance of the content of the PEQ-OHC must first be considered.
Material and methods
Development of the questionnaire
Questionnaire development was based on a literature review [Citation7], interviews with users of out-of-hours services and consultation with an expert group that was designed to ensure the content validity of the final questionnaire. The review, which identified areas of potential relevance for the measurement of user experiences with out-of-hours care, informed the content of semi-structured face-to-face interviews. The interviews were undertaken by two researchers independently with 16 users of out-of-hours care to assess which aspects of care were important for their satisfaction with out-of-hours care. There were three groups of users: first, adults over 16 years of age; second, carers who telephoned and accompanied a patient during the out-of-hours visit were included if the patient required help responding to questions; and third, parents or guardians who telephoned and accompanied children to the out-of-hours centre and completed the questionnaire on the child's behalf.
Items were devised based on the results of the review and the interviews which were presented to an expert group comprising out-of-hours staff and researchers with experience in questionnaire development and testing. The group made some minor changes to the initial list of items. The resulting questionnaire was then pre-tested in interviews with 13 users. Some small changes were made to item wording including making some items more appropriate for users coming to the centre by ambulance. Overall, the questionnaire was found to be acceptable, relevant, and understandable by users.
Data collection
A random sample of 542 users who had contacted one of three out-of-hours primary care centres in the North, West, and South of Norway from 15 April to 13 May 2008 were mailed the resulting 24-item PEQ-OHC questionnaire with a covering letter. Reminders were sent at six weeks.
The Norwegian Regional Committee for Medical Research Ethics and Norwegian Social Science Data Services approved the survey which was in accordance with the Helsinki Declaration of 1975, as revised in 1983.
Statistical analysis
Items were assessed for levels of missing data. Principal component analysis (PCA) was used to assess the underlying structure of the PEQ-OHC [Citation17]. Internal consistency was assessed using item-total correlation and Cronbach's alpha.
Construct validity was assessed through comparisons with variables previously found to be associated with patient experiences [Citation7,Citation18] and included comparisons with responses to additional questions included within the questionnaire alongside the PEQ-OHC. Access to care has been found to have an association with user satisfaction within primary care [Citation18] and is an important concern for recipients of out-of-hours care [Citation19]. It was hypothesized that PEQ-OHC scales scores would have low correlations under 0.3 with perceived difficulty of contacting the service by telephone and the amount of time taken to get to the centre. More moderate correlations in the range 0.2–0.6 were hypothesized for the self-reported waiting times at the clinic as this was more reflective of the functioning of the clinic and organizational aspects of care. Information has been found to be an important determinant of user satisfaction within primary care and more generally [Citation18,Citation20]. It was hypothesized that PEQ-OHC scores would have low to moderation correlations in the range 0.3–0.6 with perceptions of the adequacy of information relating to test results. Following previous findings moderate to high correlations in the range 0.4–0.7 were hypothesized with responses to questions relating to overall satisfaction [Citation7,Citation13,Citation21], expectations [Citation22], and incorrect treatment [Citation13]. Finally, lower PEQ-OHC scores were expected for respondents subsequently referred to their GP who need further care following their out-of-hours visit [Citation18].
Results
Data collection
Of the 542 users mailed a questionnaire, 225 (41.51%) responded. shows the respondent characteristics; 148 (65.78%) adults self-completed the questionnaire, 53 (23.56%) were parents or guardians, and 16 (7.01%) were carers. The most frequent types of contact were telephone followed by a consultation at the centre and consultation at the centre only for 125 (58.96%) and 63 (29.72%) respectively. shows that missing data ranged from 0.0% to 11.7%. The majority of items had less than 7.0% missing data.
Table I. Respondent characteristics.
Table II. Missing data, PCA loadings1 and internal consistency for the PEQ-OHC (n = 225).
Principal component analysis
The first PCA for users who had both telephone contact and a consultation at an out-of-hours centre (n = 125) gave four components with eigenvalues above 1.0 that accounted for 78.79% of the total variation (see ). All items had component loadings over 0.5. The components can be described as doctor services, telephone contact and organization, and nursing services. The second PCA for all users who had a consultation at a centre irrespective of whether they telephoned beforehand (n = 188) gave four components with eigenvalues above 1.0 that accounted for 73.98% of the total variation. The components can be described as nursing services, doctor services, organization/waiting room, and waiting times. The components of telephone contact, doctor services, and nursing services were confirmed in analyses undertaken separately for users who self-completed the questionnaire and for carers and parents or guardians completing the questionnaire (data not shown). In three of these four analyses the four items relating to organization formed a distinct component and, in all four, the question relating to unanswered questions had a relatively low component loading or was the sole item within the final weaker component.
Internal consistency
Item-total correlations for the final scales were acceptable, ranging from 0.66 to 0.89 (see ). Following consultation with the expert group, the removal of two items from each of doctor services, nursing services, and telephone contact gave three scales with a Cronbach's alpha above 0.9. The item within the telephone contact scale relating to competence had a relatively high proportion of missing data; however, following consultation with the expert group it was decided to retain this item. The removal of two items relating to information and unanswered questions from the doctor services scale increased Cronbach's alpha. The nursing services and organization items combined produced a satisfactory alpha coefficient; however, the four organization items had lower item-total correlations and their removal increased the level of alpha. Following theoretical considerations and the results of the second PCA, the organization items were treated as a separate scale. The removal of the item relating to satisfaction with the waiting area increased the Cronbach's alpha to 0.82. The final scales had similar high levels of internal consistency in separate analyses of users and carers, parents or guardians (data not shown).
Construct validity
shows that the results of 24 of the 25 tests of construct validity were in the direction hypothesized; 23 were statistically significant. PEQ-OHC telephone contact scores had a significant low level of correlation with the perceived level of difficulty in contacting the service by telephone. The three remaining scale scores were lower for users who were referred to their GP following the out-of-hours consultation with significant results for the nursing services and organization scales. These three scales also had low levels of correlation with time taken to get to the out-of-hours centre, the results being significant for the doctor services and organization scales. Scores were significantly correlated with perceived waiting times at the centre, the largest being for organization. There were significant moderate correlations for the three scale scores relating to care received at the out-of-hours centre and user perceptions of whether they received enough information regarding the results of any examination or tests. The four PEQ-OHC scale scores all had significant correlations of a generally moderate to high levels with the three global questions relating to overall satisfaction, the extent to which expectations were met, and the extent of any poor treatment at the out-of-hours centre.
Table III. Mean (SD) PEQ-OHC scores1 and correlations2 with aspects of healthcare process and global ratings (n = 225).
Discussion
Principal findings
The PEQ-OHC has undergone a rigorous process of development and evaluation based on recognized criteria for questionnaires designed to assess user experiences and satisfaction with care. The questionnaire has good evidence for data quality, internal consistency, reliability, and validity in this sample of users of out-of-hours care in Norway.
Strengths and weaknesses of the study
The PEQ-OHC has undergone more extensive development and testing than existing questionnaires [Citation7,Citation23]. The four scales had high levels of internal consistency as assessed by item-total correlation and Cronbach's alpha, the results of which were confirmed in the different groups of users. The PEQ-OHC underwent multiple tests for construct validity based on a priori hypotheses and the great majority of the results were in the hypothesized direction and were significant.
Surveys of user satisfaction usually have low response rates and 41.51% for this survey in Norway was within the range reported for the existing questionnaires used within postal surveys of user experiences with out-of-hours care, which ranged from 39.7 to 52.2% [Citation7,Citation23]. These surveys also used one reminder and further reminders might have increased the response rate. Information for the non-respondents to the Norwegian survey was not available but non-respondents to patient satisfaction surveys more generally have been found to be members of minority groups and less well educated [Citation18]. Non-respondents to surveys of users attending out-of-hours centres in the Netherlands were more likely to be female, middle-aged, and not privately insured [Citation4], while in the UK they have been found to be more likely to be younger and less affluent [Citation24]. One study in the Netherlands followed up a sample of non-respondents, concluding that overall satisfaction did not differ much for non-respondents [Citation25].
Low response rates have implications for the representativeness of data that are to be used to compare out-of-hours centres which includes national surveys. The use of further reminders and additional incentives to enhance response rates will be assessed prior to a proposed national survey of users of out-of-hours care in Norway. Respondents and non-respondents will also be compared to test for potential response bias. It was important that the present study included users who were representative of those that will be included in this national survey and hence the study followed inclusion criteria proposed for the latter. However, to the extent that further reminders and additional incentives may enhance response rates which could result in respondents with different characteristics, it is important that the PEQ-OHC is further evaluated for data quality, internal consistency, and validity in the national sample prior to the reporting of results. For purposes of the present study, a comparison group of respondents who were recruited by the same means but with additional reminders and incentives to boost response rates might increase our understanding of any response bias relating to data quality, internal consistency, and construct validity. The PEQ-OHC underwent a similar process of development to patient experiences questionnaires used in Norwegian national surveys, which show similar levels of reliability and validity in larger patient groups [Citation13–16]. Given that the highest response rate reported for a user survey of experiences with out-of-hours care was just over 10% higher than that reported here [Citation4], it is expected that a response rate that is achievable with traditional forms of reminders and incentives will have little effect on the results of questionnaire testing.
Strengths and weaknesses in relation to other studies
The systematic review [Citation7] underpinning this work identified four questionnaires for measuring user satisfaction with out-of-hours care that have not undergone such extensive evaluation [Citation4,Citation6,Citation24,Citation25]. Content validity was explicitly considered for just two of these questionnaires. Data quality was considered for just two questionnaires. PCA informed the development of three questionnaires. However, the results for important subgroups were not reported. Item-total correlation was reported for just one questionnaire and scales within two questionnaires failed to meet the 0.7 criterion for Cronbach's alpha. The four questionnaires have limited evidence for construct validity; there were no hypotheses and few of the comparisons were explicit tests of validity [Citation7].
One more recent questionnaire, the Out-of-hours Patient Questionnaire (OPQ), has undergone a more extensive evaluation than the questionnaires included in the review [Citation23]. However, users were not involved in the development of questionnaire items, which has implications for content validity [Citation7,Citation8]. Testing for construct validity was not as extensive as that for the PEQ-OHC but was based on explicit hypotheses.
Meaning of the study
There are a number of unpublished questionnaires that are being used for assessing user experiences with out-of-hours care which do not have evidence for reliability and validity [Citation7,Citation25]. The use of questionnaires such as the PEQ-OHC, which have evidence supporting their application, will improve the quality of surveys of user experiences and the use of standardized questionnaires will improve generalizability. The PEQ-OHC is short and hence acceptable for users in self-completed form making it suitable for postal surveys. The PEQ-OHC is recommended for local and national surveys of users in Norway who have received telephone advice or have had a consultation at an out-of-hours centre.
Unanswered questions and future research
Methods to enhance response rates are needed for surveys of user experiences of out-of-hours care. Future surveys should also undertake more detailed comparisons of respondents and non-respondents to more fully assess the extent of any bias. The PEQ-OHC is recommended for evaluating the effectiveness of changes in the delivery of out-of-hours care and for national surveys of users in Norway.
Acknowledgements
The authors would like to thank the members of the expert group who contributed to questionnaire and survey development, the three out-of-hours care centres (Arendal, Kvam, and Tromso) that recruited users, and Saga Hogheim for help with data collection. This study was funded by the Norwegian Knowledge Center for the Health Services.
Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
References
- Hallam L. Out-of-hours primary care. Br Med J 1997; 314:157–8.
- Hansen BL, Munck A. Out-of-hours service in Denmark: The effect of structural change. Br J Gen Pract 1998;48: 1497–9.
- Glynn LG, Byrne M, Newell J, Murphy AW. The effect of health status on patients’ satisfaction with out-of-hours care provided by a family doctor co-operative. Fam Pract 2004;21:677–83.
- Moll van Charante E, Giesen P, Mokkink H, Oort F, Grol R, Klazinga N, . Patient satisfaction with largescale out-of-hours primary health care in the Netherlands: Development of a postal questionnaire. Fam Pract 2006; 23:437–43.
- Thompson K, Parahoo K, Farrell B. An evaluation of a GP out-of-hours service: Meeting patient expectations of care. J Eval Clin Pract 2004;10:467–74.
- Salisbury C, Burgess A, Lattimer V, Heaney D, Walker J, Turnbull J, . Developing a standard short questionnaire for the assessment of patient satisfaction with out-of-hours primary care. Fam Pract 2005;22:560–9.
- Garratt AM, Danielsen K, Hunskaar S. Patient satisfaction questionnaires for primary care out-of-hours services: A systematic review. Br J Gen Pract 2007;57:741–7.
- Anden A, Andersson SO, Rudebeck CE. Concepts underlying outcome measures in studies of consultations in general practice. Scand J Prim Health Care 2006;24:218–23.
- Evans RG, Edwards A, Evans S, Elwyn B, Elwyn G. Assessing the practising physician using patient surveys: A systematic review of instruments and feedback methods. Fam Pract 2007;24:117–27.
- Vedsted P, Sokolowski, Hanne NH. Data quality and confirmatory factor analysis of the Danish EUROPEP questionnaire on patient evaluation of general practice. Scand J Prim Health Care 2008;26:174–80.
- Sitzia J. How valid and reliable are patient satisfaction data? An analysis of 195 studies. Int J Qual Health Care 1999;11:319–28.
- Department of Health. National quality requirements in the delivery of out-of-hours services. Gateway no. 3776. London: Department of Health; 2004. 1–8.
- Garratt AM, Andersen Bjertnæs Ø, Krogstad U, Gulbrandsen P. The OutPatient Experiences Questionnaire: Reliability and validity in 52 Norwegian hospitals. Qual Saf Health Care 2005;14:433–7.
- Garratt AM, Bjørngård JH, Dahle KA, Bjertnæs ØA, Saunes IS, Ruud T. Psychiatric Out-Patient Experiences Questionnaire: Data quality, reliability and validity in patients attending 90 Norwegian clinics. Nordic J Psychiatry 2006;60: 89–96.
- Danielsen K, Garratt AM, Andresen Bjertnæs ØA, Petersen KI. Patient experiences in relation to health care process and provider characteristics. Scand J Pub Health 2007;35:70–7.
- Garratt AM, Andresen Bjertnæs ØA, Barlinn JK. Parent Experiences of Paediatric Care (PEPC) questonnaire: Reliability and validity following a national survey. Acta Paediatr 2007;96:246–52.
- Hair J, Anderson RE, Tatham RL, Black WC. Factor analysis. Multivariate data analysis with readings. 4th ed. Englewood Cliffs, NJ: Prentice-Hall; 1995. 364–419.
- Crow R, Gage H, Hampsom S, Hart J, Kimber A, Storey L, . The measurement of satisfaction with healthcare: Implications for practice from a systematic review of the literature. Health Technology Assess 2002;6.
- Richards SH, Pound P, Dickens A, Greco M, Campbell JL. Exploring users’ experiences of accessing out-of-hours primary medical care services. Qual Saf Health Care 2007;16:469–77.
- Fitzpatrick R. Patients’ assessments of the outcomes of primary care consultations. Scand J Prim Health Care 1993;11(Suppl 2):68–71.
- Jenkinson C, Coulter A, Bruster S, Chandola T. Patients’ experiences and satisfaction with health care: Results of a questionnaire study of specific aspects of care. Qual Saf Health Care 2002;11:335–9.
- Jackson JL, Chamberlin J, Kroenke K. Predictors of patient satisfaction. Soc Sci Med 2001;52:609–20.
- Campbell JL, Dickens A, Richards SH, Pound P, Greco M, Bower P. Capturing users experiences of UK primary out-of-hours primary medical care: Piloting and psychometric properties of the Out-of-hours Patient Questionnaire. Qual Saf Health Care 2007;16:462–8.
- McKinley RK, Manku-Scott T, Hastings AM, French DP, Baker R. Reliability and validity of a new measure of patient satisfaction with out-of-hours primary medical care in the United Kingdom: Development of a patient questionnaire. BMJ 1997;314:193–8.
- Van Uden CJ, Ament AJ, Homba SO, Zweitering P, Crebolder HFJM. Patient satisfaction with out-of-hours primary care in the Netherlands. BMC Health Serv Res 2005;5:6.