The Impact of Multisource Feedback on Continuing Medical Education, Clinical Performance and Patient Experience: Innovation in a Child and Adolescent Mental Health Service

ABSTRACT

This paper reiterates the importance of the role of multisource feedback (MSF) in continuing medical education/continuing professional development (CME/CPD) and its impact on doctors’ performance and patient experience globally. It summarises a unique initiative of robust utilisation of internationally recognised multisource feedback tools in an outpatient child and adolescent mental health service (CAMHS) in Qatar. The process involved the effective adoption and administering of the General Medical Council’s (GMC) self-assessment questionnaire (SQ), patient questionnaire (PQ), and colleague questionnaire (CQ) followed by the successful incorporation of these tools in CME/CPD. The original version of the PQ questionnaire and the instructions to the patient document were translated into Arabic through the blind back-translation technique. This initiative of introducing gold-standard MSF tools and processes into clinical practice, among other quality-improvement projects, has contributed to the improvement of service standards and doctors’ clinical practice. Patient satisfaction was measured through the annual patient experience analysis using the Experience of Service Questionnaire (ESQ) whereas changes in doctors’ performance were evaluated by comparing annual appraisal scores before and after implementation of this initiative. We have demonstrated that when MSF is obtained impartially and transparently using recognised and valid tools, it can improve patient experience and enhance doctors’ performance

KEYWORDS:

• Continuing medical education
• continuing professional development
• multisource feedback
• appraisal
• colleague feedback
• patient feedback

Introduction

There has been an increasing interest in recent times in the evaluation of the clinical practice of doctors and its role in the process of appraisal and revalidation [1]. Doctors must regularly demonstrate that they are providing good care to their patients and keeping up to date through continuing medical education/continuing professional development (CME/CPD) in order to maintain their medical licence. An important component of this process is the acquisition of feedback about their performance which is intended to guide learning and performance change. Multisource feedback (MSF) refers to the gathering of feedback from patients and colleagues using reliable and structured questionnaires to support the continued professional development of clinicians [2,3]. It is considered an integral part of the medical licencing renewal and revalidation process by the regulatory authorities globally as well as an important aspect of trainees’ evaluation during postgraduate training [2,4,5]. The role of MSF in providing trustworthy information towards the appraisals of other clinical disciplines including nurses is also established [6].

The process of acquiring MSF also ensures that the preferences and values of patients are included in designing and delivering services resulting in improvement of patient experience. When applied appropriately using recognised and valid tools/questionnaires, MSF has the potential to enhance the quality of care provided by healthcare services. The General Medical Council (GMC) in the United Kingdom has developed its own colleague and patient feedback tools to assess the professional performance of its doctors. These tools were based on its principles of “Good Medical Practice” [7]. Soon after introduction, their reliability, validity, and utility were explored [8]. The three questionnaires, Self-assessment Questionnaire (SQ), Patient Questionnaire (PQ) and Colleague Questionnaire (CQ) have undergone further development and have demonstrated their psychometric properties consistently [9,10].

In this paper, we describe a novel CME initiative involving the adoption of the GMC’s MSF questionnaires to assess the clinical performance of doctors in the child and adolescent mental health service (CAMHS) at Hamad Medical Corporation (HMC), in Qatar. It is a community-based multidisciplinary outpatient service providing support to children and young people with mental and behavioural disorders and their families [11]. We also report how the tools (PQ and instruction document) were modified through a robust process of Arabic translation to suit the needs of the local patient population and the impact of the initiative on patient satisfaction levels. This project of effective adoption and application of the gold standard GMC MSF tools in the HMC CAMHS service in Qatar, to our knowledge, is a pioneering achievement in CME in the Middle East and North Africa (MENA) region. There is no previous published literature related to the applicability of MSF tools in the MENA region and we hope this innovative project will encourage further investigation into the crucial component of CME regionally.

Aim of the Initiative

We aimed to improve clinical practice, patient experience, and CME/CPD standards through creating awareness among doctors regarding the importance of obtaining reliable feedback about their clinical performance using valid and recognised tools. The aim of the project was consistent with HMC’s current strategic plan and in alignment with Qatar’s National Health Strategy 2018–2022. The national strategic enablers required to achieve excellence in clinical care include introducing innovation, leading-edge best practices and new models of care, implementing evidence-based practice, and delivering patient and family-focused care. This initiative endorsed the International Patient Safety Goal (ISPG) of improving effective communication with patients and their families as outlined by the Joint Commission International (JCI). While there were existing forms and methods to obtain MSF across the different training programs in HMC, we felt the process needed further enhancement to ensure its effectiveness. The implementation of this initiative was approved by the leadership of the mental health service, HMC.

Methods

a) Development of appropriate tools

The questionnaires to obtain feedback from colleagues and patients were adopted from the ones that are currently in use by the GMC, UK, after obtaining necessary permissions. The GMC has developed a Patient Questionnaire (PQ) (Table 1) a Colleague Questionnaire (CQ) (Table 1), and a self-assessment questionnaire (SQ) (Table 2) for this purpose. Each questionnaire takes 5 to 10 minutes to complete and includes core items (rated on a five-point scale) relating to the doctor’s performance, global assessment items (rated on a binary scale), a free-text comment box, and items that collect contextual/demographic information about the respondent.

Table 1. Wording and response categories for items on the patient questionnaire (PQ) and colleague questionnaire (CQ).

Patient Questionnaire (PQ)
	CORE ITEMS		CONTEXTUAL & DESCRIPTIVE ITEMS
Q4a	Being polite	Q1	Who is filling the patient questionnaire?
Q4b	Making you feel at ease in his/her presence	Q2	Reason(s) why the patient saw the doctor
Q4c	Listening to you	Q3	How important the visit was to the patient’s health/ well-being?
Q4d	Assessing your condition	Q8	Was the patient’s visit with their usual doctor?
Q4e	Explaining your condition and treatment	Q10	Patient’s gender
Q4f	Involving you in decisions about your treatment	Q11	Patient’s age group
Q4g	Providing or arranging treatment for you	Q12	Patient’s ethnic group
Q5a	This doctor will keep information about me confidential		OTHER ITEMS
Q5b	This doctor is honest and trustworthy	Q9	Free text comments box: Please feel free to add any other comments you have about this doctor.
	GLOBAL ASSESSMENT ITEMS
Q6	I am confident about this doctor’s ability to provide care
Q7	I would be completely happy to see this doctor again
Colleague Questionnaire (CQ)
	CORE ITEMS		GLOBAL ASSESSMENT ITEM
Q1	Clinical knowledge	Q19	This doctor is fit to practice medicine.
Q2	Diagnosis
Q3	Clinical decision making		CONTEXTUAL & DESCRIPTIVE ITEMS
Q4	Treatment (including practical procedures)	Q21	Colleague’s gender
Q5	Prescribing	Q22	Colleague’s age group
Q6	Medical Record Keeping	Q23	Professional role
Q7	Recognising and working within own limitations	Q24	How recently familiar with the doctor’s clinical practice
Q8	Keeping knowledge and skills up to date	Q25	Frequency of contact with the doctor
Q9	Reviewing and reflecting on own performance	Q26	Colleague’s ethnic group
Q10	Teaching (students, trainees, others)
Q11	Supervising colleagues		OTHER ITEMS
Q12	Commitment to care and well-being of patients	Q20	Free text comments box: Please feel free to add any other comments you have about this doctor.
Q13	Communication with patients and relatives
Q14	Working effectively with colleagues
Q15	Effective time management
Q16	This doctor respects patient confidentiality
Q17	This doctor is honest and trustworthy
Q18	This doctor’s performance is not impaired by ill health

Table 2. Wording and response categories for items on the Self-assessment questionnaire.

Self-assessment Questionnaire (SQ)
	CORE ITEMS		GLOBAL ASSESSMENT ITEM
Q1a	Clinical knowledge	Q5	I am fit to practise medicine.
Q1b	Diagnosis
Q1c	Clinical decision making		CONTEXTUAL & DESCRIPTIVE ITEMS
Q1d	Treatment (including practical procedures)	Q2	Does your current role include direct consultations with patients?
Q1e	Prescribing
Q1f	Medical Record Keeping
Q1g	Recognising and working within own limitations
Q1h	Keeping knowledge and skills up to date
Q1i	Reviewing and reflecting on own performance
Q1j	Teaching (students, trainees, others)
Q1k	Supervising colleagues		OTHER ITEMS
Q1l	Commitment to care and well-being of patients	Q6	Free text comments box: Please add any other comments you want to make about your own performance.
Q1m	Communication with patients and relatives
Q1n	Working effectively with colleagues
Q1o	Effective time management
Q3a	Being polite to patients
Q3b	Making patients feel at ease
Q3c	Listening to patients
Q3d	Assessing patients’ medical condition
Q3e	Explaining patients’ condition and treatment
Q3f	Involving patients in decisions about their treatment
Q3g	Providing or arranging treatment for patients
Q4a	I respect patient confidentiality
Q4b	I am honest and trustworthy
Q4c	My performance is not impaired by ill health

The original version of the PQ questionnaire along with the instructions document was translated into Arabic in order to meet the needs of the local patient population effectively. The process of translation was facilitated by a panel of psychiatrists and psychiatric nurses, all having excellent proficiency in both English and Arabic Languages, using blind back-translation technique. The translated versions were reviewed to ensure that the original and the translated versions achieved conceptual, cultural, and linguistic equivalence. The translated documents were piloted on the first five respondents. Their understanding of each item on the questionnaire and its response was explored to ensure the translated items retained the same meaning as the original items. The colleague and self-assessment questionnaires were not translated as all staff working in the service were fluent in English.

We used the Experience of Service Questionnaire (ESQ) [12], developed by the Health Care Commission, UK to measure the impact of this novel initiative on patient satisfaction. The likely impact on doctors’ performance was assessed through the change in their annual appraisal scores.

b) Process and implementation

The process of obtaining MSF in HMC CAMHS starts with doctors being evaluated receiving the self-assessment questionnaire (SQ) from the senior executive secretary for completion. The SQ [Table 2] allows them to compare their own perceptions about their professional performance to the views of their patients and colleagues. It comprises 26 core items and maps on to the content of the PQ (7 items) and the CQ (19 items) [Table 1]. Ten other items collect background information about the doctor and the context in which they practice. The doctor being assessed, after responding to all the items, provides the completed SQ to the senior executive secretary in CAMHS along with a list of names of 20 colleagues in the team nominated by the doctor to be approached for colleague feedback.

The child and adolescent mental health service at HMC comprises of psychiatrists, psychiatric nurses, psychologists, speech and language therapists, occupational therapists, administration staff, a dietitian, and a social worker (Figure 1). When selecting colleagues, the expectation is to include at least one senior peer, one junior peer, line manager, a member each from all disciplines in the MDT including administration staff. The senior executive secretary provides a copy of the CQ with instructions on how to complete the questionnaire to the nominated colleagues. The responses are chased until a minimum of 15 completed responses are received confidentially. All responses are anonymised to ensure transparency and avoid bias.

Figure 1. MDT composition of HMC CAMHS.

The senior executive secretary is also responsible for collecting feedback from a minimum of 25 patients, using PQ. The forms are offered to consecutive patients booked in the doctor’s clinic until the required 25 responses are received. The completed questionnaires have no patient identifiable information and are returned confidentially. The final report (summary of responses) is then shared with the doctor being evaluated and is reviewed in detail during the annual appraisal with their clinical supervisor. Doctors do not see any individual responses from colleagues or patients. Once all responses are received, the senior executive secretary compiles the outcome of all the three forms (SQ, CQ and PQ) for each doctor into a summary report and shares it confidentially with the respective appraiser. The summary report contains the scores for each item across the three questionnaires and therefore allows for a comparison between the responses of the doctor, colleague and the patient.

Results

a) Patient satisfaction

We measured patient satisfaction in the annual patient experience analysis using the Experience of Service Questionnaire (ESQ), developed by the Health Care Commission, UK. The findings were compared against a similar analysis done in 2017 to see if the implementation of this project since 2018 had induced any change.

A total of 140 participants took part in the service evaluation exercise in 2019 compared to 94 in 2017. The respondents were divided into three groups: a) children aged 9–11 years b) young people aged 12–18 years and c) Parents/carers, as required by the ESQ. The ESQ consists of 12 items rated “Certainly True” = 1, “Partly True” = 2, “Not True” = 3, and a “Don’t know” option.

The rate of satisfaction among parents/carers increased from 94% in 2017 to 100% in 2019. Similarly, a noticeable improvement was noticed in children aged 9–11 years as the rate increased from 75% to 90%. There was no change noticed in the 12–18 years age group as all expressed satisfaction except one adolescent choosing to remain neutral in each survey. It was noted that the proportion of children, young people, and parents who responded either “not true” (dissatisfaction) or “don’t know” to the question seeking their view about the quality of overall care received, had reduced from 4.25% of the total participants in 2017 to only 1.4% in 2019.

It is pertinent to note here that the contribution of MSF per se towards the improvement in the standard of a service is hard to quantify. The improvement in service standard in this project is therefore likely to be an outcome of multiple quality improvement initiatives including the introduction of reliable and gold-standard MSF.

b) Doctors’ performance

The positive impact has also been demonstrated in the annual appraisal of the doctors working in HMC CAMHS. A considerable improvement was noticed in individual practice and overall scores from 2017 to 2021 (most recent appraisals). All doctors being appraised improved their annual appraisal scores by at least one point on the scale ranging from 1(does not meet expectations) to 5 (Excellent). The minimum score obtained was 4 (exceeds expectations). Doctors have been able to reflect on the feedback received and put measures in place to further improve the care provided to children and young people. Furthermore, this project has enhanced further the teamwork between members of the multidisciplinary team thereby increasing the productivity of the entire team.

Future Directions

Feedback from colleagues can help with better integration and communication between members of the team. There is previous published literature concluding that doctors who engage in MSF demonstrated a significant improvement in several professional behaviours [13]. There is a need for replication of this project, both locally and regionally, to further improve care standards across different clinical disciplines and services. The role of effective MSF in the continued professional development of doctors and improvement of healthcare service delivery needs more recognition globally. There is a need in the future for more work to be undertaken in clinical services in the middle eastern region focusing on adaptation and incorporation of globally recognised MSF tools to enhance performance of doctors and improvement of quality of healthcare services.

Conclusion

Multisource feedback, when obtained impartially and transparently using recognised and valid tools, has the potential to transform clinical practice, patient experience, and CME/CPD standards. The acquisition of patient and colleague feedback allows clinicians to reflect on deficiencies in their clinical practice to enable them to improve clinical performance and achieve excellence in care. The implementation of gold-standard MSF tools as part of the CME/CPD process has the potential to contribute to the improvement of overall patient care and service standards globally.

Acknowledgments

We would like to acknowledge Dr. Basim Samir, Dr. Saleem al Nuaimi, Dr. Manal Osman, Ms. Asieh Dehwari, Mr. Mahmoud Alshamlawi, Mr. Lazarus Phiri, Ms. Sheena Shaheer and Dr. May Jasem Almeraisi for their support. Open Access funding provided by the Qatar National Library.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This project received no specific grant from any funding agency, commercial or not-for-profit sectors.