Validation of the Italian version of the Patient Reported Experience Measures for intermediate care services

Abstract

Background

Intermediate care (IC) services are a key component of integrated care for elderly people, providing a link between hospital and home through provision of rehabilitation and health and social care. The Patient Reported Experience Measures (PREMs) are designed to measure user experience of care in IC settings.

Objective

To examine the feasibility and the scaling properties of the Italian version of PREMs questionnaires for use in IC services.

Methods

A cross-sectional survey was conducted on consecutive users of 1 home-based and 4 bed-based IC services in Emilia-Romagna (Italy). The main outcome measure was the PREMs questionnaire results. PREMs for each home- and bed-based IC services were translated, back-translated, and adapted through consensus among the members of the advisory board and pilot testing of face validity in 15 patients. A total of 199 questionnaires were returned from users of bed-based services and 185 were returned by mail from users of home-based services. The return rates and responses were examined. Mokken analysis was used to examine the scaling properties of the PREMs.

Results

Analysis performed on the bed-based PREMs (N=154) revealed that 13 items measured the same construct and formed a moderate-strength scale (Loevinger H=0.488) with good reliability (Cronbach’s alpha =0.843). Analysis of home-based PREMs (N=134 records) revealed that 15 items constituted a strong scale (Loevinger H=0.543) with good reliability (Cronbach’s alpha =0.875).

Conclusion

The Italian versions of the bed- and home-based IC-PREMs questionnaires proved to be valid and reliable tools to assess patients’ experience of care. Future plans include monitoring user experience over time in the same facilities and in other Italian IC settings for between-service benchmarking.

Keywords:

• intermediate care
• patient-reported experience measures
• validation
• elderly

Plain language summary

The number of elderly people with multiple chronic conditions is increasing, as a result of the aging of the Italian population. Health and social services provided by a coordinated multi-professional staff in semi-residential facilities or at patient’s home (the so-called intermediate care [IC] services) are one of the organizational responses to the emerging needs of these people. This study was conducted to examine patients’ experience of care with IC services, using a questionnaire originally developed in English. The questionnaire was translated into Italian, revised by the working group to improve the wording of items, and then administered to a small group of patients to test its comprehensibility. The final form of the Patient Reported Experience Measures (PREMs) questionnaires, with incorporated changes, was then administered to patients treated at home and in community hospitals. The questionnaires were well understood by patients and were easy to administer by the health care staff. The large majority of patients reported a positive experience of care. After a statistical analysis, 2 items were omitted from the bed-based version of the PREMs. The final versions of the PREMs proved to be valid and reliable.

Introduction

The prevalence of long-term conditions such as heart disease, stroke, cancer, type 2 diabetes, obesity, and arthritis is increasing globally.¹ Moreover, the presence of multimorbidity (the co-occurrence of at least two chronic conditions in one individual) has become common, especially in elderly people, ranging from 26% to >50% in patients aged 65–69 years and from 56.6% to 80% in patients aged ≥85 years.^2,3 Although clinical guidelines and care pathways have been developed for the management of specific chronic conditions, the treatment of multimorbidity requires a holistic approach to address patients’ complex health care needs and to ensure continuity of care.^4–7

IC is one of the organizational responses to emerging needs of elderly people with multiple chronic diseases. While in the UK IC services are more set up for rehabilitation/reablement, in Italy the term IC denotes health and social services provided by a coordinated multiprofessional staff in semi-residential facilities or at patient’s home. IC is aimed at preventing unnecessary hospitalizations and delayed discharge, facilitating discharge home, developing clinical pathways alternative to hospital and based on patients’ care needs, promoting access to rehabilitation services and physical therapy, and reducing long-term institutionalization.^8,9

The UK National Audit of Intermediate Care (NAIC), assisted by the Plain English Campaign, defined IC as:

services [that] offer a link between places such as hospitals and people’s homes, and between different areas of the health and social care system – community services, hospitals, GPs and social care.¹⁰

In the UK, four types of IC have been defined: crisis response, bed based, home based, and reablement.¹⁰

In Italy, the National Health Plans 2011–2013 and 2014–2016 have redesigned the health care network by strengthening community care and integrating it with hospital care. This reorganization of health care services has included the development of IC services.¹¹

IC bed-based services, the so-called community hospitals (CH), possess the following characteristics: 15 beds, managed by nurses with health care provided by a general practitioner (or other physicians). Emilia-Romagna has been a pioneer in this process, implementing 14 bed-based CH services since the 1990s.

Home-based services provide care in patients’ home, including rehabilitation, supportive and technical nursing care, domestic aid, and personal care (as well as respite care provided to informal caregivers).¹² In the Emilia-Romagna region, home-based services have been implemented successfully for several decades.¹³

Fremont et al¹⁴ argued that the optimal management of chronic diseases relies on the degree of patient concordance with medications and therapeutic interventions inside and outside the hospital. Concordance is more likely if patients are aware of their condition and understand the treatment options available. Assessment of patients’ experience is thus considered a key element in this new model as it is recognized as a fundamental indicator of quality health provision.

Experience and satisfaction are different constructs. Experience measures whether specific processes of care did, or did not, occur. Several studies have shown that improving patients’ experience is often associated with increase in patient safety, clinical effectiveness, and health outcomes.^15,16 However, patient satisfaction with care may not correlate with processes of care or clinical outcomes. Patients may be satisfied with poor quality care or dissatisfied in a highly effective service.¹⁷ Integration of measures of patient experience into evaluation of health care performance identifies problem areas (poorly delivered processes of care) that may be addressed to improve the quality of delivered care. The Patient Reported Experience Measures (PREMs) offer an objective and reliable measure of users experience and a practical way to explore person-centered, integrated care models.

There is an increased emphasis in Europe on measures of patient experience as health care quality indicators across different medical disciplines. PREMs have been widely used and integrated in the evaluation of quality of care for chronic heart failure,¹⁸ mental health,¹⁷ diabetes, asthma, and rheumatoid arthritis,¹⁹ in emergency care services,²⁰ and to measure patients’ experience with medicine use.²¹ The NAIC in England and Wales has included PREMs designed specifically for IC since 2013.^22,10

Creating and validating patient-reported indicators should be a priority, especially in areas where care is complex and demand is increasing, such as in case of chronic condition management. A wide and complete assessment is necessary to describe and improve care pathways.

In IC settings, positive personal experience represents a strong contribution to the process of recovery. Given the potential relevance of this measurement in IC, we have translated and adapted the English IC-PREMs²³ to the Italian health care setting to evaluate patient experience in 4 bed-based and 1 home-based IC settings in the Emilia-Romagna region (Italy). The specific aim of this study was to examine the feasibility, acceptability, and scaling properties of the Italian version of IC-PREMs (hereafter IC-PREM-I) questionnaires (available on request from the corresponding author).

Methods

The original versions of the PREMs questionnaires developed by Teale et al²³ and used in NAIC 2015 were translated by one of the authors and back-translated by a bilingual person of British origin.

Some questions relevant to both bed- and home-based services feature in both questionnaires (eg, patient involvement in defining treatment aims or sharing health information with patient’s family); other questions are service specific (eg, agreement about appointment times or how to contact staff in case of need in home-based services). Both questionnaires contain an open-ended narrative response question. A question regarding proxy completion was included with both questionnaires.

The study protocol was submitted to the Local Ethics Committees (Comitato Etico Interaziendale Bologna Imola and Comitato Provinciale di Modena) and the study qualified as exempt from approval because questionnaires were anonymous and research activities presented no psychological and/or physical risk to human subjects. A form explaining the study procedures was given to patients. All patients provided oral informed consent. Those not willing to participate did not fill the questionnaires.

Bed-based IC-PREM-I

The translated questionnaire was reviewed by the advisory board of Castelfranco Emilia and Castel San Pietro Terme health districts (2 of the 4 CHs involved in the validation project), and suggestions for possible modifications were discussed, reviewed, and incorporated into the final version. This final form was first administered in a pilot phase at Castel San Pietro’s CH, where professional nurses were instructed about the administration and then conducted a structured interview to 15 patients to evaluate feasibility and face validity. The bed-based version of PREMs questionnaire was subsequently administered to patients admitted to 4 CH of Emilia-Romagna region (Italy) voluntarily involved in the project (Castel San Pietro, Castelfranco, Forlimpopoli, and Fanano). Patient completion of the questionnaires occurred 1–2 days prior to discharge, supported by professional nurses. Data collection occurred between January 1st and September 30th of 2016.

Home-based IC-PREM-I

After the forward–backward translation of the questionnaire, a working group of Bologna Local Health Authority reviewed and modified some items to improve their comprehension. The home-based final version of PREMs questionnaire was administered to 500 patients receiving home care in the metropolitan area of Bologna, Emilia-Romagna (Italy), between April 1st and May 31st of 2016. The questionnaire was handed over to the patients with a prepaid envelope at the end of the service. Patients were asked to return the envelope by surface mail.

Statistical analysis

Item responses were recoded to a dichotomous format using the approach suggested by the Picker Europe²⁴ and adopted by Teale et al in the IC-PREM validation paper.²³ Specifically, responses indicating a problem were scored as “0” and responses indicating a good experience were scored as “1”.

Mokken scale analysis was used to analyze the construct validity of the PREMs. It is based on nonparametric item response theory (IRT) models and consists of an automated item selection procedure (AISP), which partitions items into one or more scales, possibly leaving some items unselected.²⁵

The goodness-of-fit of the IRT model was assessed by computing the Loevinger H-coefficient (measuring scalability) and Cronbach’s alpha coefficient (measuring the reliability of the scale). Mokken proposed the following rules of thumb for H – a scale is considered weak if 0.3≤H<0.4, moderate if 0.4≤H<0.5, and strong if H>0.5.³⁰ Cronbach’s coefficient is considered good if alpha is ≥0.8. Valid Mokken analysis rests on meeting a number of assumptions: unidimensionality, local independence, and latent monotonicity.

We planned to administer at least 150 questionnaires in each setting in order to ensure adequate item classification.²⁶ R software (Mokken package) was used to conduct all the analyses.

Results

IC-PREM-I bed-based questionnaires results and Mokken analysis

Of the 199 bed-based questionnaires administered, 32.2% were filled in by patients, 42.2% by caregivers, and 10.1% by patients and caregivers. For 15.6% of the questionnaires, information on the type of respondent was not available. Overall, 154 questionnaires (77%) were returned complete and were used for statistical analysis. The item response frequency is provided in Table 1. Item responses were largely positive and ranged from 65% to 97%.

Table 1 Percentage of positive responses to the bed-based questionnaire

Question number	Items	Percentage of positive responses
Q1	Staff have sufficient information	97.4
Q2	I was given enough information	87.7
Q3	Aware of goals	91.6
Q4	Involvement in goal setting	81.2
Q5	Involvement in decisions about care/treatment	71.4
Q6	Family or carer involved	74.0
Q7	Questions answered	89.6
Q8	Confidence in staff	90.3
Q9	I knew the coordinator	94.2
Q10	Involved in discharge decision	81.8
Q11	Home circumstances considered	75.3
Q12	Information provided for family	85.7
Q13	Treated with dignity and respect	90.9
Q14	Ability in maintaining social contacts improved	64.9
Q15	Information about other services available	75.3

Mokken analysis of the 15-item questionnaire revealed that items Q2 (“I was given enough information”) and Q15 (“information about other services available”) had Loevinger H-coefficients <0.3, and the overall scalability was weak (H=0.379, standard error [SE] =0.051). Using AISP, one latent trait was identified. Items Q2 (“staff have sufficient information”) and Q14 (“ability in maintaining social contacts improved”) were discarded because they did not measure this trait. After removing these items, the Loevinger H-coefficient improved (H=0.448, SE =0.058) and Cronbach’s alpha was 0.843. The monotonicity assumption and local independence assumption were met; therefore, the final scale consisted of 13 items (Table 1).

IC-PREM-I home-based questionnaires and Mokken analysis

The postal return rate of IC-PREM-I home-based questionnaires was 37%. Of the 185 questionnaires received, 29.1% were filled in by patients and 68.7% by caregivers. For 2.2% of the questionnaires, information on the respondents who filled in the details was not available. Overall, 134 questionnaires (72%) were returned complete and used for statistical analysis. The item response frequency is provided in Table 2. Item responses were largely positive and ranged from 66% to 98%.

Table 2 Percentage of positive responses to the home-based questionnaire

Question number	Items	Percentage of positive responses
Q1	Length of wait time is reasonable	97.8
Q2	Staff have sufficient information	92.5
Q3	Aware of goals	91.0
Q4	Involvement in goal setting	73.1
Q5	Involvement in decisions about care/treatment	79.9
Q6	Aware of how to contact staff	97.0
Q7	Visit time convenient	83.6
Q8	Questions answered	88.8
Q9	Confidence in staff	92.5
Q10	Involved in decision about end of home care	75.4
Q11	Given enough notice about end of home care	66.4
Q12	Information provided for family	79.9
Q13	Discussion regarding further services after end of home care	92.5
Q14	Treated with dignity and respect	96.3
Q15	Ability in maintaining social contacts improved	79.1

Results of Mokken analysis indicated that all items had Loevinger H-coefficients >0.3 and the overall scalability was H=0.543 (SE =0.065). Again, using the AISP, only one latent variable was identified and all items measured this trait. The monotonicity assumption and local independence were met, and the final scale consisted of all 15 items (Table 2; Cronbach’s alpha =0.875).

Areas of improvement in IC in Italy

Response rates were largely positive for all items and across questionnaires. Using an arbitrary cutoff of 75% to define a moderate-to-good level of satisfaction with care, some key areas were identified for targeted improvement:

• In the hospital setting: involvement in decisions about care, family or caregiver involvement in treatment, and maintenance of social contacts.

• In the home setting: involvement in goal setting and clear information to patients about when the service would end.

Opinions, complaints, and positive comments derived from the open-ended question have been used to create word clouds to provide easy visual insight about patients and care experience (Figure S1).

Discussion

The IC-PREM-I bed- and home-based questionnaires are valid and reliable tools to assess patients’ experience in IC settings. The questionnaires were well understood by users and were easy to administer by the health care staff. Mokken analysis showed that the bed- and home-based final versions, consisting of 13 and 15 items, respectively, measure a single robust construct, allowing summation of scores to give an overall measure of experience.

The pilot administration of the questionnaires identified some areas for service improvement: better involvement of patients in decisions about their care and their ability to maintain social contacts in hospital facilities and need to improve information provision regarding the end of services. These aspects of IC require the attention of policy makers and IC staff to improve patients’ experience of care and to inform service development at both organizational and clinical levels.

As patient-reported measures are used to drive quality of care improvement, the Italian validation also offers the possibility to use the questionnaire for benchmarking, to compare different IC facilities, as emphasized by the recent recommendations about patient-reported indicators provided by the Organisation for Economic Co-operation and Development.²⁷

To our knowledge, no other validated tools are available in Italy to assess patients’ experience of their care. We provided a new and useful tool for IC services to be integrated with other health care quality indicators to progress toward person-centered care.

The positive response rate to the items of the bed-based IC-PREM-I was very high in the present study compared with 2015 NAIC data.¹⁰ In contrast, patients reported a less positive experience of involvement in taking decisions regarding their care as measured by the home-based IC-PREM-I when compared with 2015 NAIC data. While these could be true phenomena, social desirability bias^28,29 should be considered as a possible explanation; patients wish to be viewed positively by health care assistants and staff for their expectations to be met. Patients may be more willing to report a less favorable experience of care through a postal survey than they are when nursing staff supports completion.

This study has some limitations. The questionnaires have been administered in a region in Italy where IC facilities are long established (Emilia-Romagna). It is possible that administering PREMs in other Italian regions would have yielded different results. The postal questionnaires (home-based services) had a relatively low return rate (37%), though this is comparable with other studies using postal questionnaires¹⁵ and higher than the return rate in the original validation study (12.6%).²³

Future plans are to continue the administration of PREMs in the facilities involved in the present study, to monitor user experience over time, and to determine whether changes at organizational level in IC facilities and services are responsive to patient need. We intend to involve further IC facilities in the periodical survey to allow regional comparison and benchmarking.

Acknowledgments

The authors wish to thank the colleagues of the working groups of Bologna Local Health Authority and of the community hospitals for their support in the conduction of the study: Mara Morini, Luca Barbieri, Laura Biagetti, Juri Coccia, Loretta Muraro, Manuela Petroni, Chiara Petrucci, Claudio Tacconi, Giulia Pieri, Vittorio Donini, Sabrina Gabrielli, Luigi Baldi, Loretta Vallicelli, Antonella Dallari, Stefano Boni, Andrea Spanò, and Alessandra Raggi.

Disclosure

The authors report no conflicts of interest in this work.