Development and validation of an instrument to measure patient engagement in Hong Kong Special Administrative Region, China

Abstract

Objective

The objective of this study was to develop, pilot test, and psychometrically validate a patient engagement questionnaire, called Patient Engagement Index (PEI), in Hong Kong, China.

Method

The method employed was based on a patient engagement framework and literature review and expert panel discussion. A new measure named PEI with 20 items divided into five factors was developed. A pilot study of 40 patients was used to confirm the clarity of PEI. Explored factor analysis was used to confirm the construct validity. Cronbach’s alpha, intercorrelation coefficients (ICCs), and weighted kappa were used to assess the internal consistency reliability and test–retest reliability separately. Rasch model and differential item functioning were also used to further confirm the fit of the instrument.

Results

In all, 324 participants successfully completed the survey. Based on the evaluation of exploratory factor analysis and theoretically considerations, a four-factor structure comprising 20 items was identified. Rasch analysis also confirmed that the model has a good fit (ranging from 0.662 to 1.294). All domains were considered internally consistent (alpha >0.7). Test–retest showed both ICC (>0.6) and weighted kappa (>0.4) meeting the minimum recommended standard. No ceiling or floor effect was found. Furthermore, Spearman’s correlation coefficient confirmed that PEI had good internal validity.

Conclusion

PEI was shown to have good reliability and validity. This is a short, quick, and appropriate tool to assess the extent of patient engagement in both clinical and research settings.

Keywords:

• patient engagement
• questionnaire development
• psychometric property
• quantitative research
• evaluation
• Hong Kong

Introduction

Patient engagement is considered the cornerstone for building the foundation of patient-centered care,^1,2 a radical reform for the current health care system³ and the next “blockbuster drug of the century.”⁴ In recent years, despite a growing number of policy makers, health care providers, and academics who tirelessly advocated that patient engagement should be a major focus of health care policy efforts and reform, substantial changes were rarely noticed in clinical practice.^1,5

Patient engagement became an important issue in the field of health care, mainly attributed to the change in disease epidemiology from infectious to chronic.⁶ This change informed the medical professionals that the old paradigm of the health care system gradually fails to relieve patients’ pain, discomfort, or depression and even increases the number of patients with chronic conditions who would rather spend most of their time taking care of themselves at home than in the hospital.⁷ This understanding holds despite these professionals’ expertise to offer a few of the most specialized, technically advanced treatments to patients. Patients, families, and even communities are required to play an equally important and active role as professionals in maintaining and promoting health.^8–10 Moreover, evidence has indicated that patients who are actively engaged in their health care could experience safer, more effective, and less expensive health care services than their passive counterparts.^11,12 When patients were involved in the process of shared decision making, they demonstrated high satisfaction and medication adherence, improved quality of life, and healthy lifestyles.^13–15

Although the potential benefits of patient engagement have been largely recognized by a growing body of studies,^11–15 rapid changes and developments of supplies and demands of health care services cause significant challenges and difficulties to engaged patients in the health care system.¹⁶ The key question, that is, how to engage patients in health and health care and a series of outstretched questions, such as which aspects encourage or impede patients to engage in health care, continue to explicitly or implicitly confuse regulators and academics.^1,17 In Hong Kong, 75% of the people aged ≥65 years and 45% of the people between 45 and 64 years are suffering from at least one or more chronic diseases.¹⁸ Patients with chronic or latent chronic conditions requiring long-term ongoing care and consuming a large portion of health care resources should be encouraged to play an active role in maintaining their own functioning, which is a priority for controlling and managing chronic conditions.¹⁹

Although patient engagement has been advocated as an essential driver for promoting efficiency, improving outcome, and decreasing cost, several disputes and confusions remain. The concepts and definitions of patient engagement are unclear,^20,21 the synonymous terms are ambiguously used in assessing engagement,¹⁷ and few empirical studies focus on measuring engagement.²² Carman et al indicated that without reliable and valid instruments to assess the performance of patient engagement in health care, policy makers have difficulty understanding the performance of the health care system from the patients’ perspective; moreover, a lack of empirical evidence exists to establish the baseline of engagement to compare the effectiveness and efficiency of different interventions toward patient-centered care improvement.¹

Very few measurement tools exist to help identify patients’ engagement.^1,16,17 Several survey instruments only measure patient experience or certain synonymous concepts, such as involvement,²³ activation,¹⁹ or psychometric variations,¹⁷ or variables from the general population’s perspective,¹⁶ not the patient’s. Currently, few available instruments adequately address the different levels of engagement that evaluate the patient–professional engaged relationship.

In recent years, despite the Hong Kong policy makers’ continuing encouragement that patients can actively engage in their own health care, evaluating the effectiveness of any substantial change in the culture, quality, or service pattern of health care service is difficult. The lack of a suitable instrument to assess engagement is the main obstacle.²⁴ Therefore, this paper describes the development of the Patient Engagement Index (PEI), a new measure of patient engagement among outpatients grounded in rigorous conceptualization and appropriate psychometric methods from both international experiences and local perspectives.

Method

Questionnaire development

PEI was designed to be a generic instrument with components evaluating patient engagement in a general outpatient setting in Hong Kong. It is an instrument to evaluate the level of patient engagement from a holistic perspective which could overcome the shortcomings of previous instruments that could only evaluate some concepts synonymous with engagement. The draft of PEI was developed based on the modified conceptual framework, literature reviews, expert panel discussion, and patient feedback. The original framework was established based on literature reviews and a series of previous focus group interviews of both professionals and patients in the local health care system.²⁵ The modified framework suggested that patient engagement should be a relationship, which is reciprocal, dynamic, and pluralistic, rather than an invariable status among patients, professionals, and the organizations. The framework presented five levels of patient engagement (self-information search, enquiry, bidirectional communication, discussion, and partnership). Each level represented a certain extent of patient engagement. The extent of patient engagement slides on this relationship axis back-and-forth according to the variation of patients’ psychological, physical, and socioeconomic conditions. By reviewing relevant literature and available instruments and based on the framework, an initial set of 33 potential items of PEI was developed. After a three-round discussion among a committee of experts with multiple service backgrounds in the health care system (a physician, a registered nurse, a hospital administrator, a health care researcher, and a patient), a final version of 20 items was developed. The PEI assesses patient engagement in four phases of the entire health care journey, including “diagnosis,” “treatment,” “rehabilitation,” and “health maintenance,” as suggested by the expert panel and considered from the findings of previous studies.^15,26,27 Based on the design of previous instruments that aimed to assess patient experience and our experts’ discussion, a 5-point Likert rating scale was adopted in this instrument from “never” to “always.” A convenient sample of 10 people with experience of using local health care services was randomly recruited to test the readability and intelligibility of the questionnaire. A group of 40 patients (10% of the total sample) from a general outpatient clinic (GOPC) was then recruited to participate in the pilot study to verify the interview guide, the questioning skills, the availability of the recruitment plan, and the arrangement of fieldwork.²⁸ Based on the feedback, certain items were reworded and this subsequently resulted in a final version of 20 items for the following psychometric evaluations.

Validity study

Study design

Trained interviewers conducted a cross-sectional validation survey based on the face-to-face interview, depending on the patient’s medical condition and request in one of the biggest GOPCs in Hong Kong.

Participants

Inclusion criteria included 1) local language speaker (Cantonese), 2) aged ≥18 years, 3) must be an established patient, and 4) should be able to understand the interview (as judged by the interviewer). A minimum of 300 eligible patients were the target for this study, which was confirmed to satisfy the requirement of sample size to conduct the psychometric evaluations (exploratory factor analysis [EFA] and confirmatory factor analysis) in this study.²⁹ The reasons for choosing participants from a GOPC include 1) patients with physical fitness could answer the questionnaire, 2) Hong Kong patients were highly encouraged to visit a GOPC first when they required medical consultations, and 3) GOPC mainly provides services to patients with stable chronic conditions.

Patients who satisfied the inclusion criteria were invited to attend a face-to-face interview using the structured questionnaire by well-trained investigators. All the respondents were required to sign the written consent form before the interview. The interviewers explained the purpose, the process, the potential risks and conflicts, the confidentiality, and contact point of the survey to the participants. Moreover, the respondents were informed of their right to withdraw from participating in the survey at any time.

Statistical analysis

R (R Foundation, Vienna, Austria; version 3.3.1) was employed to manage data and perform statistical analysis in our study. Respondents with over 10% missing values were removed from the analysis dataset.³⁰ The data were double checked to ensure accuracy. The demographics of the respondents and the response profiles were presented descriptively. For psychometric evaluations, reasonability, construct and content validity, internal consistency, and test–retest reliability and Rasch analysis were included.

Reasonability was judged mainly by three factors, namely, the response rate, time to complete the questionnaire, and variation of the item. Response rate not only means the ratio of people who agreed to participate in the survey but also the ratio of people who properly answered each item of the questionnaire. A low response rate could lead to missing values and could damage the quality of data. Time was considered to assess the length of the survey. A short questionnaire may damage the validity and reliability,³¹ but a long questionnaire could increase the response burden and lower the quality of data. Ceiling and floor effect have been used to test the variation of items.

EFA with oblique rotation (oblimin) was performed to confirm the construct validity. Both Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity were conducted to confirm the appropriateness of conducting EFA. The KMO statistic varies between 0 and 1.0. Values >0.5 are considered “barely acceptable,” and >0.9 are deemed “superb.”³² For Bartlett’s test, a significant statistic (P≤0.05) means it can efficiently perform a factor analysis on the dataset.³³ Pairwise deletion of missing values, eigenvalues >1.0, and factor loading score ≥0.4 were adapted to sort items into factors. In case of multiple loading >0.4 of an item on several factors, the item was determined to be located in the factor with greater conceptual relationship.³⁴ An overarching item was set to test the internal validity of PEI. Content validity was checked through expert panel and patient feedback.

Internal consistency reliability and test–retest reliability were used to check whether the items, which tested the concept, were the exact items to be tested. Cronbach’s alpha was utilized to test internal consistency, which ranges from 0 to 1.0. Streiner et al considered an alpha value of >0.7 as acceptable.³⁵ Items with a low corrected item-total correlation (r<0.2) were considered to be removed from the scale.³⁶ Test–retest reliability, with a 2-week interval, was used to assess instrument consistency. Pairs for each item were evaluated using a weighted kappa coefficient and for sub-scale total scores that were analyzed using intercorrelation coefficient (ICC). Values >0.4 for weighted kappa and 0.6 for ICC were accepted as satisfactory levels for reliability.³⁵

Rasch model was also utilized in the study to test the fit of the model. Employing the partial credit model was suggested, which allowed items to have polytomous response categories and did not require the distance between the thresholds of response to be the same.¹⁶ The fit statistic between 0.5° and 1.5° was considered as satisfactory.³⁷ Furthermore, differential item functioning (DIF) was also checked for possible item bias caused by responses from different subgroups (gender and chronic disease in this study) in the sample.

Furthermore, the weights for each item were calculated using factor analysis. Then, the overall and domain scores were transferred using min–max normalization ranging from 0 to 100. A higher score indicated better engagement. These scores were calculated using the following formula: ${\mathrm{z}}^{\prime }=\frac{\mathrm{z}-\min (\mathrm{x})}{\max (\mathrm{x})-\min (\mathrm{x})}$where z is the raw score, min (x) indicates the minimum value of the attribute, and max (x) indicates the maximum value of the attribute.

Ethics approval

Ethics approval was obtained from the Joint Chinese University of Hong Kong–New Territories East Cluster Clinical Research Ethics Committee. All the personal details have been carefully stored and kept confidentially.

Results

Table 1 presents the characteristics of the participants. A total of 324 participants were successfully recruited. The response rate was 53.7%. Among the respondents, 54% were male, >80% were older than 40 years, <10% lived alone or in the institute, and 82.1% reported living with at least one kind of chronic condition.

Table 1 Participant demographics and frequencies

Indicator	Participants
	n	%
Gender
Male	175	54.0
Female	149	46.0
Age group (years)
18–40	53	16.4
41–60	150	46.3
≥61	121	37.3
Education
No education/primary	53	16.4
Secondary	109	33.6
Post-secondary	93	28.7
Tertiary or above	69	21.3
Received government allowance
Yes	33	10.2
No	291	89.8
Current living status
Living alone	16	4.9
Living with family/others	305	94.1
Living in institution^Citation2	3	1.0
Current working status
Retired	110	34.0
Unemployed	11	3.4
Employed	142	43.9
Housewife	56	17.3
Full time student	5	1.4
Having chronic condition
Yes	266	82.1
No	58	17.9

The KMO and Bartlett’s test confirmed that all 20 items were well intercorrelated (r=0.90, P<0.001). Using the EFA, four factors were extracted, which explained 64% of the total variance. Although the loading of item 7 (0.37) was <0.4, its good content validity prompted us to include it in the PEI for further assessments. Patients (99%) expressed that they had no difficulties or embarrassment to understand and answer any of the questions of PEI. Combined with the suggestions from expert’s discussion, 20 items under four dimensions were grouped for further analysis (Table 2).

Table 2 Exploratory factor analysis, reliability analysis, and Rasch analysis of the 20-item PEI

No	Questionnaire items	EFA				Internal consistency			Test–retest		Rasch analysis		Internal validity
		Dimensions				Item-total correlation	Cronbach’s alpha	Alpha if item deleted	Weighted kappa	ICC	Outfit	Infit
		1	2	3	4
	Overall						0.91			0.82			0.61a
	Dimension I: self-information search						0.79			0.71			0.40a
1	Search diagnosis information about your health problem (1)	0.78				0.67		0.62	0.87		1.208	1.207
2	Search treatment information about your health problem (eg, medicine, surgery) (6)	0.77				0.72		0.67	0.64		1.294	1.206
3	Search self-care information about your health problem (11)	0.74				0.77		0.59	0.59		1.033	1.040
	Dimension II: communication with professionals						0.73			0.75			0.48a
4	Ask medical staffs diagnosis information about your health problem on your own initiative (2)		0.40			0.39		0.32	0.68		0.971	0.931
5	Ask medical staffs treatment information about your health problem on your own initiative (eg, medicine, surgery) (7)		0.37			0.48		0.42	0.63		0.923	0.846
6	Ask medical staffs self-care information about your health problem on your own initiative (12)		0.57			0.68		0.59	0.41		1.268	1.217
7	Require the medical staff’s explanation about the diagnosis results of your health problem (3)		0.74			0.54		0.45	0.68		1.119	1.051
8	Require the medical staff’s explanation about the treatment process of your disease (8)		0.63			0.68		0.59	0.80		0.937	0.969
9	Require the medical staff’s explanation about how to improve the ability of self-care (13)		0.57			0.66		0.54	0.50		0.699	0.681
	Dimension III: partnership						0.90			0.67			0.50a
10	Share your experience and perspective with medical staff to discuss the diagnosis results (4)			0.63		0.66		0.63	0.62		0.916	0.860
11	Share your experience and perspective with medical staff to			0.72		0.80		0.76	0.45		0.952	0.967
	discuss the treatment regimen (9)
12	Share your experience and perspective with medical staff to discuss how to improve the ability of self-care (14)			0.67		0.79		0.74	0.79		0.879	0.919
13	To partner with medical staff and decide to make disease management plan together (5)			0.81		0.75		0.71	0.73		0.880	0.886
14	To partner with medical staff and make a decision to choose and implement the treatment regimen together (10)			0.86		0.79		0.75	0.55		0.689	0.718
15	To partner with medical staff and make a decision to relieve the symptoms and recapture health together (15)			0.83		0.81		0.76	0.50		0.767	0.778
	Dimension IV: health maintenance						0.82			0.64			0.52a
16	Search the information to maintain your physical and mental health (16)				0.51	0.43		0.40	0.60		1.264	1.225
17	Ask medical staffs information about how to maintain physical and mental health on your own initiative (17)				0.77	0.72		0.66	0.61		1.022	1.023
18	Require the medical staff’s explanation about how to maintain physical and mental health (18)				0.82	0.75		0.67	0.40		1.086	1.069
19	Share your experience and perspective with medical staff to discuss how to maintain physical and mental health (19)				0.50	0.81		0.71	0.70		0.662	0.716
20	To partner with medical staff and make a decision to maintain physical and mental health together (20)				0.49	0.78		0.68	0.63		0.752	0.748

Note:

a P<0.01. Numbers in parenthesis indicate the original order of items in the questionnaire.

Abbreviations: PEI, Patient Engagement Index; ICC, intercorrelation coefficient; EFA, the Exploratory Factor Analysis.

For reliability, Cronbach’s alpha of the overall scale was 0.91, and those of the dimensions ranged from 0.73 to 0.90. All corrected item-total correlations were >0.4. In addition, all the values of alpha if the item was deleted were lower than the value of alpha for each dimension, which means that the internal consistency of the PEI was satisfactory. A sample of 30 participants was retested after 2 weeks interval. The results of both ICC and weighted kappa met the criterion, indicating that PEI has good test–retest reliability. Rasch analysis employed infit and outfit statistics to measure the item fit. They ranged from 0.681 to 1.294, which are all within the acceptable range. For internal validity, Spearman correlation coefficients for the overall (r=0.61, P<0.001) and each dimension (r=0.40–0.52) were statistically significant and generally satisfactory (Table 2).

Table 3 lists the item-level descriptive statistics, with the normalized mean and median, indicating no ceiling and floor effect for the overall (1.34% vs 29.92%) and for each dimension. The floor and ceiling effects corroborated the degrees of satisfaction observed in the mean scores of 47.83 and 26.89 for the issues of partnership and communication with professionals, respectively.

Table 3 Descriptive statistics for each dimension

Dimension	Overall	Self-information search	Communication with professionals	Partnership	Health maintenance
n	324	324	324	324	324
Range	0–100	0–100	0–100	0–100	0–100
Min	4.17	0	4.17	0	0
Max	93.06	100	100	95.83	100
Median	37.5	41.67	50	50	30
Mean	37.6	38.09	47.83	26.89	29.01
SD	16.87	23.16	15.94	22.04	19.91
Ceiling effect (%)	1.34	1.52	1.83	0.98	1.22
Floor effect (%)	29.92	21.92	17.87	47.13	30.33

Discussion

The scarcity of a parsimonious and robust instrument to assess patient engagement resulted in the lack of evidence-based interventions necessary for drawing the attention of professionals and policy makers to encourage patient engagement.¹ Several factors influence whether and to what extent patients and professionals should develop an engaged relationship to keep health at different levels and at different points along the journey.³⁸ To date, although instruments (eg, the Patient Activation Measure [PAM] or the Patient Health Engagement scale [PHEs]) were developed to measure the active role of patients in their care, they appear incapable of capturing the complexity and dynamism of the patient–professional engaged relationship in health care. In particular, PAM was developed based on activation, which was defined by the National Health Service as describing the knowledge, skills, and confidence a person has in managing his/her own health and care.³⁹ PAM is highly related to the patients’ behaviors and attitude to participate in health care.¹⁷ Although PHEs was explored specifically on engagement, it only discovered the emotional elaboration of the disease and only assessed the psychological effect on patients’ lives.¹⁷ On the contrary, PEI discovered the patient–professional engaged relationship, which evaluates the level at which a patient and professional interact with each other. Carman et al indicated that patient engagement should be a broad term to describe that both patients and professionals played equally important roles in the engaged relationship to promote health care.¹ Achieving a satisfactory patient–professional relationship is difficult without efforts from both patients and professionals. Thus, from our perspective, the path toward reaching a distinct level of patient–professional relationship in the different phases of the health care journey must be considered to understand patient engagement.

The PEI was structured based on previous qualitative studies, systematic analysis of the literature, multiple rounds of expert discussions, and a series of rigorous psychometric evaluations in the local healthcare system. For reasonability, the 20-item PEI required an administration time of nearly 3–5 minutes, which was lower than an unacceptable time of 15 minutes.⁴⁰ The percentage of missing data (0.1%–0.2%) was very low in the survey, which means that all the respondents could understand the items of PEI very well. For reliability, PEI yielded a very good Cronbach’s alpha value >0.9, and for each domain, the value is also very satisfactory ranging from 0.73 to 0.90.³⁴ The PEI provided acceptable to excellent test–retest reliabilities with ICCs between 0.64 and 0.82, and weighted kappa ranged between 0.41 and 0.87.³⁵ Regarding validity, the EFA yielded four factors, which suggested the unidimensionality of the scale and good fit of the model with the data. Moreover, the Rasch model further confirmed the theoretical model of the PEI scale and the importance in terms of fitting each item of the scale. No DIF items were found in PEI. For internal validity, the Spearman correlation confidence for the overall scale was 0.61, which indicated a good causal relationship.⁴⁰

Generally, the 20-item instrument evaluated patient engagement at different phases of the entire health care journey. The transferred easy-to-understand summary score (0–100) can be used to compare the overall level of engagement or the performance at each phase. This validation focused on a sample of patients from the outpatient setting. Thus, further research will be necessary to validate the PEI in various settings, such as inpatient or accident and emergency departments. Moreover, patient feedback confirmed the good content and face validity of PEI and helped us better understand their feelings, patient engagement in the local health care system, and their preferred method of expressing their values, preference, and needs in clinical encounters.

Interestingly, other than the first three domains of PEI reflecting the different levels of patient engagement, health maintenance was discovered as another single dimension, which covered all the different relationships between patient and professional engagements. This finding potentially indicated that the patients’ mindset, priorities, and communication of engagement may be totally different at daily health management (health maintenance) from their disease management (diagnosis, treatment, and rehab). This finding inspired us to explore and evaluate further the patients’ engagement from the perspectives of both disease and health under different lenses. The interventions to engage patients in medical services and live in the community might be totally different, which were never mentioned in previous studies. A win or lose situation of the policy to promote engagement should be based on whether patients play an appropriate role at the appropriate time with an appropriate method of communicating with the health care system and using health or social resources in maintaining health.¹

The other creativity lies in a well-constructed PEI managing complexity by identifying the deepest concerns and values of the patients’ perception toward engagement at the different phases of the entire health care journey. An individual with a high PEI score does not necessarily indicate an effective performance in all the phases of health care.⁴¹ For example, extremely well engaged at diagnosis period but averagely engaged at treatment period may result in a high overall score of PEI. The application of PEI could illustrate the variation of the patient–professional engaged relationship along the progress of health care. Such examples could help clinical leaders and policy makers to quantify these variations. Moreover, they could wisely arrange the limited health and social resources to promote the patient and professional’s engaged willingness and provide effective services to improve the health outcomes.

Although the assessment of hospital performance is increasingly considering patient experience, health care providers and academics are still raising strong questions on how to assess patient engagement and what engagement methods to use.^42–44 A local study found that a patient who displays partial involvement in decision making implied a better quality of life than those who fully or do not involve in decision making.⁶ This finding implied that dealing with patient engagement as a one-size-fits-all strategy for everyone at every time might not be suitable. Therefore, as a well-developed instrument based on international experience and local evidence to grasp the patient’s direct experiences of health care at different levels of the different phases rather than merely a general feeling, PEI appeared to be valuable both theoretically and empirically.

Inevitably, a few limitations exist. Firstly, the current PEI only has a traditional Chinese version, which limited its application. A simple Chinese or English version will be developed in the near future. Secondly, only patients from one GOPC were recruited in our study. Generalization might be a problem, and further analyses in larger population samples should be conducted to further confirm the reliability and validity of this evidence. Finally, the results may be limited by the nature of this cross-sectional study.

Practice implications

The 20-item PEI is a brief and easily answerable instrument to assess patient engagement. This instrument can be applied in both research and clinical settings to understand patient preferences, priorities, and relationship with professionals at the different phases of health care. For researchers, the measure assesses the patient’s extent of engagement and establishes the connection between patient engagements and other interventions or experimental manipulations to control the cost, promote efficiency, and improve health outcomes. For clinical practice, the measure can help professionals design the targeted health care plan according to patient’s preference, values, and needs, to identify and provide services in a more preferred and effective method, and finally improve the health outcomes.

Conclusion

The PEI is the first validated tool in Hong Kong or China that examines patient engagement. This measurement has been proven to be reliable and consistent in helping professionals, administrators, and policy makers to gain further understanding of whether hospital services have positively engaged patients in health care and put patients at the center of every decision. The findings provide important insights into developing interventions for improving the quality of care and laying the foundation for further research on patient-centered care.

Disclosure

The authors report no conflicts of interest in this work.