Improving the quality of clinical training in the workplace: implementing formative assessment visits

Abstract

Family physicians have a key role to play in strengthening district health services in South Africa. There are a number of barriers to the supply of these specialists in family medicine, one of which is the quality of workplace-based training and low pass rate in the national exit examination. The South African Academy of Family Physicians in collaboration with the Royal College of General Practitioners has adopted a short course to train clinical trainers and a process of formative assessment visits (FAVs) for clinical trainers in the workplace. Training programmes have struggled to implement the FAVs and this article reports on the experience at Stellenbosch University and the issues identified. Clinical trainers who participated in FAVs mostly set developmental goals for themselves that focused on improving the learning environment and consolidating personal skills in training and assessment. The FAVs were beneficial for the family physician trainers, their managers and the academic family physicians at the university. The tools and process for conducting the FAVs may be of value to other programmes.

Keywords::

• family medicine
• formative assessment
• medical education
• primary care
• quality assurance
• workplace-based assessment
• workplace-based training

Introduction

Family physicians have a key role to play in achieving South Africa’s health reforms and policy goals.¹ National health insurance and universal health coverage rely not only on access to care, but also the quality of that care. Family physicians, with additional postgraduate training to become specialists in family medicine, can improve the quality of primary and district level care, when they are part of healthcare teams.^2,3 Family physicians accomplish this through their roles as clinicians, consultants, capacity builders, clinical trainers, leaders of clinical governance and supporters of community-orientated primary care.⁴ The medium-term goal is to have a family physician at every district hospital, community health centre and sub-district in South Africa.⁴

One of the barriers to achieving this goal is the low pass rate in the national examination and inadequate supply of new family physicians coming out of the nine training programmes.⁵ Other barriers include the creation and funding of registrar and family physician posts (career pathing), inappropriate placement in referral hospitals and lack of proper recognition in the private sector. Strategies to improve the pass rate in the national examination include more attention to selection of registrars and better workplace-based clinical training and assessment.

Many of the family physicians involved in clinical training of registrars are not adequately prepared for this role. Older training programmes did not include attention to the skills required as a trainer or capacity builder. National learning outcomes for family physicians now include skills in teaching and training,⁶ but programmes may still vary considerably in how they achieve these outcomes. Workplace-based training for specialists is also a new activity within the district health services. This means that managers and other healthcare workers may not be used to support postgraduate training or in creating a suitable educational environment.

In response to this situation the discipline of family medicine in South Africa has created a short five-day course for the training of clinical trainers (TCT). The TCT was originally developed in a collaboration between Stellenbosch University and the Royal College of General Practitioners that was funded by the European Union and Tropical Health Education Trust. Subsequently the initiative has been taken over by the South African Academy of Family Physicians, which offers the course to all programmes on an annual basis. To date more than 100 family physicians have completed the course in South Africa from all training programmes. The TCT has also been offered to other countries in the region through a project entitled Family Medicine Leadership Education and Assessment Programme (FaM-LEAP).⁷

The TCT covers a range of competencies needed for workplace-based training and assessment in five key areas: ensuring safe and effective patient care through training, establishing and maintaining an environment for learning, teaching and facilitating learning, enhancing learning through assessment, and supporting and monitoring educational progress.⁸

Evaluation of the TCT suggests that participants shift their capability significantly after the course in these five key areas.⁹ The effect of the TCT, however, appears to diminish over time and by six months students of those who participated could no longer detect a change.⁹

These findings suggested that in addition to the TCT there was a need for an ongoing process of support and formative development of clinical trainers.⁸

Development of the visit

At the end of the TCT participants were expected to create a set of developmental goals and activities for themselves over the next six months. A follow-up visit to the clinical trainer in his/her workplace was envisaged as a way of sustaining the new skills, reinforcing these goals and offering suggestions for further change.

The visit was designed to be supportive and formative in nature and to enable ongoing development of the trainer. The visit could then be repeated annually to ensure incremental improvement in the quality of clinical training. A structured approach to the visit was developed using a number of tools:

1. Tour the facility in order to understand the educational context and environment;

2. Meet with and receive feedback from the registrars on their experience of training;

3. Observe the family physician training the registrars;

4. Obtain the family physician’s self-evaluation of effectiveness as a clinical trainer using a validated tool;

5. Provide feedback to the family physician, discuss and create an action plan;

6. Provide feedback to the facility managers and discussion of any issues.

Senior family physicians from each academic department were trained to perform these visits. Following the training they accompanied one of the TCT facilitators on a visit led by the facilitator. The local family physician then conducted another visit that he/she led with support and feedback from the facilitator. Most of the participants in the initial TCT project received at least one such formative visit after the course.

Subsequently, however, most of the training programmes failed to sustain the initiative, and feedback at the Academy’s National Education and Training Committee suggested that these visits had not become institutionalised. At Stellenbosch University we also struggled to implement the visits in a sustainable manner. Training-complex coordinators were often performing outreach to training facilities, but did not incorporate the more formal assessment and creation of development plans into their visits. Feedback from these coordinators suggested that the visit manual was too long and overly complicated and that observing actual training was not always possible. There was an impression that the visit required too much preparation and would take too long. Despite agreeing to perform the annual visits none of the training complex coordinators managed to do so. Coordinators may also have been too close to the clinical trainers and the local context to offer constructive feedback, and achieve sufficient objectivity and critical awareness.

Eventually the formative assessment visits were successfully implemented by the programme coordinator and head of division from the university. This meant visiting 11 district hospitals in the Western Cape once a year. A visit took on average half a day and the visit manual was streamlined to one page of instructions and associated tools for interviewing the registrars, observing training (if possible), the trainer’s self-evaluation and co-development of a summary with specific goals and actions (see Supporting Information).

Feedback from the visits

The issues identified at these visits in 2018 along with the specific developmental goals and actions planned by the clinical trainers were extracted from the summaries of each visit. Altogether 35 goals were identified from visits to clinical trainers at nine different training sites. These goals were grouped according to the five key areas of the effective clinical trainer as shown in Figure 1. Goals were mostly set in the domains related to establishing an environment for learning as well as skills for teaching, learning and assessment. Ensuring the needs of patients were met while performing clinical training was not identified as an issue and monitoring and evaluating progress was also not a major focus area.

Figure 1: Frequency of developmental actions per specific area of expertise for the clinical trainer.

The specific actions that were identified in each of these domains are listed in Table 1.

Table 1: Actions identified to develop clinical trainers

Domain	Specific actions identified
Supporting and monitoring the educational process	Supervisor to understand the curriculum and university structures/modules
	Supervisor to regularly check performance in e-portfolio and encourage its use by registrars
Enhancing learning through assessment	Do more planned observations of clinical skills and give structured feedback
	Support and help the registrar to prepare for the exit exam
	Support and monitor the educational process; help registrars to accept feedback
	Organise direct/indirect supervision at the PHC clinics
Teaching and facilitating learning	Help registrars to develop their learning plan and identify their learning needs, particularly their skills for leadership and clinical governance
	Focus on developing senior registrar skills as apprentice family physician by giving responsibility across all roles
	Use a range of teaching methods for clinical skills
	Link each registrar to a specific supervisor who takes responsibility
	Help registrars with their research project
Establish and maintain an environment for learning	Ensure a one-hour weekly structured and protected training time, and attend the registrar meetings
	Build capacity of the multidisciplinary team members to also assess and provide feedback
	Support the registrar to be involved with outreach specialists regularly
Ensure safe and effective patient care through training	None

Reflections on visits

These initial visits created a strong foundation to assess whether the required educational standards were met and to enable continuous improvement of teaching.¹⁰ The required educational standards are not yet fully articulated in South Africa. There is consensus on the national learning outcomes,⁶ the required clinical skills¹¹ and workplace-based training and assessment as per the national portfolio of learning requirements.¹² The portfolio implies what is expected and the role of the clinical trainer has been outlined,¹³ but detailed educational competencies are still to be defined. The roles of the effective clinical trainer in the self-assessment questionnaire were adapted from a UK tool.¹⁴

The visits focused on the educational issues and not the quality of care and health services. The quality of care and range of services are of course key aspects of the learning environment and the ability of the site to support learning. Differentiating between training sites on the basis of quality of care is not common or possible in South Africa as such standardised data are not readily available. The ideal clinic initiative does provide some assessment of norms and standards in primary care, but the focus is more on the inputs than on service delivery.¹⁵

The academic family physicians, based at the university, found the visits beneficial in understanding the different trainer-related issues and contextual factors at each training site. Training sites differed considerably, and this allowed a more textured and nuanced approach to be developed. Best practices could also be identified and shared between sites.¹⁰ The training sites reported that the visits from the faculty made them feel more connected and supported.

Triangulation of information from the registrars, trainer, university and facility managers added to the credibility of the findings.¹⁶ The questionnaire tool allowed effective feedback to be given as it stated specific criteria and identified the gap between current performance and what was regarded as effective or excellent. Feedback is intended to help recipients plan how they will reduce this gap, but identifying and affirming good educational practice is also important.¹⁶ Immediate and appropriate feedback from a respected source is effective and therefore the seniority of the visitor may be important.^16,17 In our case the visitor was the head of division or the programme coordinator from the university.¹⁶

Keeping the instructions and forms brief and practical helped to frame the visit as personal and supportive rather than administrative or bureaucratic.¹⁶ The visits took 3–4 hours, which is similar to the mean reported elsewhere.¹⁷ The qualities of the visitor may be important and might include skills in listening, assessment and giving effective feedback.¹⁶ Recording of the visit in a simple one- to two-page form that summarised the overall findings of the visit to the facility as well as personal goals and actions for the trainer was also important. Such records allowed clear agreement on what needed to be done, and enabled communication with other stakeholders and follow-up at the next visit.

For the clinical trainers the visits led to a clear set of goals and actions for the year. These could also be incorporated into their own performance appraisals with their supervisors in the Department of Health and, where applicable, the university. Trainers found the clarification of developmental goals helpful in terms of focusing their energies on a few key goals. Clinical trainers on the distributed platform appear to welcome feedback on their performance and appreciate input from recognised mentors.¹⁸

Key tips for conducting a visit have been identified and resonate with the experience of the visitors in South Africa:

• Allocate time to prepare for the visit and ensure the relevant staff are available;

• Encourage self-assessment prior to the visit;

• Raise awareness of the curriculum and educational standards;

• Engage a variety of staff and not just the clinical trainer;

• Ensure feedback is clearly given and recorded.¹⁰

Self-assessment has been highlighted as a particularly useful part of the visit.¹⁰

While the university had regular interaction with the clinical trainers, the relationship with facility managers was more tenuous and these visits helped to build relationships and clarify expectations. Many issues identified in the visit related to the organisational and educational environment more than the individual clinical trainer’s competence. Involving the facility managers was therefore important.

At the district level the visits also enabled clearer feedback to directors and chief directors on the challenges facing the training programme. In many instances this elicited support and action from the directors to strengthen clinical training. Managers were committed to developing family physicians fit for purpose.

Ideally all training programmes should adopt such a process of regular FAVs with constructive feedback and action plans. The goals set during the FAVs suggest that workplace-based clinical training is not yet embedded in the work of family physicians. The emphasis was on trying to create an environment for learning within health services that have not historically included postgraduate training, and which are burdened by high service demands. At the same time clinical trainers were trying to develop appropriate educational practice.

Stellenbosch University has also pioneered the development of a learning portfolio for registrars to document learning and for clinical trainers to document workplace-based assessment (WPBA).^12,19,20 More recently, an e-portfolio has been introduced and this allows the postgraduate programme coordinator to monitor the engagement of registrars and clinical trainers easily. The FAVs also included feedback on utilisation of the portfolio as an indicator of workplace-based learning, training and assessment. In order to rely on WPBA the capability of clinical trainers and the quality of the portfolio need to be assured.²¹

Future directions

Currently we plan annual FAVs to clinical trainers, which is congruent with the need to consolidate the development of clinical training sites. In certain cases we revisited after six months, as there were issues critical to the success of clinical training. However, as sites and trainers become both more numerous and better established it may also be possible to visit less frequently, for example every 2–3 years.

This open-forum article reports on our experience with the FAVs, but further evaluation and exploration through research may be helpful; for example, exploring the barriers and facilitators to implementation of FAVs in different programmes across the country. It may also be possible to explore approaches to the direct and indirect (via audiotape or videotape) observation of training with feedback during these visits and whether this adds value. Watching a videotaped teaching session is common in UK practice visits.¹⁷

Quality improvement of family medicine training should ultimately become more formalised and may require a team of trained visitors.¹⁰ The National Education and Training Committee have decided to monitor coverage of the TCT and subsequent FAVs across the programmes going forward. They also plan to reinforce the process through offering training at the annual National Family Practitioners Conference. It is also possible that the Health Professions Council of South Africa could include questions on how clinical trainers are supported and developed during their accreditation visits. Ultimately a system of more formal recognition of competency as a clinical trainer may be needed.

Trainers may be interested in supportive networks¹⁸ and the South African Academy of Family Physicians has set up a special interest group that could potentially develop into such a network of clinical trainers.

Conclusion

This article shares a process for quality improvement of workplace-based training in family medicine that could be adopted by all postgraduate training programmes. The process has arisen from the Training of Clinical Trainers course and could be monitored by the South African Academy of Family Physicians’ National Education and Training Committee.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 Acknowledgements: This document is based on the Nov 2014 version of the NHS Pan London Quality and Regulation Unit’s General Practice Educator Application Form and the 2-yearly educator self-reporting form.

Supporting Information

Annual formative assessment visit to a clinical trainer

The purpose of the visit is to support the development of a family physician clinical trainer in the workplace. The visit should be conducted by a senior family physician who is familiar with the process of formative assessment and giving feedback. The visit should have the following components:

1. Introduction and welcome. The visiting family physician should be introduced to key people at the facility, such as the facility manager, and have a tour of the facility in order to understand the context.

2. Reflection with the visiting family physician on their current performance as a clinical trainer/supervisor in the workplace. This should use the six self-evaluation questions in Appendix A. The questions can be discussed and then completed during the visit. If there is a previous developmental plan (i.e. from the training of clinical trainers course or formative assessment visit) this should also be reviewed.

3. Feedback from registrars that you are supervising. The visiting family physician should meet separately with a representative sample of the registrars and elicit feedback on the training site, quality of training and supervision. The interview guide in Appendix B can be used to structure this conversation.

4. Observation of clinical training. The visiting family physician should directly observe clinical training during their visit or if this is not possible listen to an audio or look at a video recording of the supervisor conducting clinical training or education (maximum 10 minutes). Appendix C can be used to help observe, assess and provide useful feedback.

5. Feedback, discussion and agreement on a developmental plan for the clinical trainer. The visiting family physician should provide feedback on the visit as a whole. Achievements and strengths should be celebrated. The issues raised should be discussed with the clinical trainer and key developmental goals for the next year should be elicited and agreed on. How the clinical trainer will go about attaining these goals and how the academic department can assist should also be discussed. The SMART goals and planned actions by different people to achieve the goals should be documented in Appendix D. This summary should be retained by the clinical trainer, the visiting family physician and shared with the Head of Department. The goals and plan should be revisited over the year during interactions with the visiting family physician to monitor and encourage progress.

It should be possible to complete the visit over 3–4 hours. Another option is to complete the components of the process over a number of visits.

Appendix A: Self-evaluation of role as clinical trainer¹

Please consider your current work as a supervisor or clinical trainer of registrars and evaluate yourself according to the Likert scale and definitions given below for an effective supervisor.

• Struggling to be effective—fulfils almost none of the criteria;

• Partly effective—fulfils less than half of the criteria;

• Mostly effective—fulfils more than half of the criteria;

• Fully effective—fulfils all the criteria for an effective supervisor;

• Excellent—in addition meets most of the criteria for an excellent supervisor.

1. Ensuring safe and effective patient care through training

Table

The effective supervisor	The excellent supervisor also
▪ Balances the needs of service delivery with education ▪ Acts to ensure the health, well-being and safety of patients at all times ▪ Ensures that registrars have undertaken appropriate induction at the facility ▪ Allows registrars, when suitably competent, to take responsibility for care, appropriate to the needs of the patient ▪ Ensures that registrars understand the importance of providing culturally competent care	▪ Uses educational interventions to enhance patient care ▪ Involves registrars in service improvement ▪ Involves patients as educators
	Notes on why this rating was given:

Read the criteria above for effective and excellent supervisors and then rate yourself as:

Table

Struggling to be effective

Partly effective

Mostly effective

Fully effective

Excellent

2. Establishing and maintaining an environment for learning

Table

The effective supervisor	The excellent supervisor also
▪ Encourages participation through provision of equality of opportunity and acknowledgement of diversity ▪ Ensures that registrars receive the necessary instruction and protection in situations that might expose them to risk ▪ Encourages and maintains the confidence of registrars; is open, approachable and available ▪ Maintains good interpersonal relationships with registrars and colleagues ▪ Provides protected time for teaching and learning ▪ Involves the multiprofessional team in the delivery of teaching and supervision ▪ Is aware of the team’s experience and skills relating to teaching and supervision ▪ Ensures that workload requirements on registrars are legal and that, wherever possible, they do not compromise learning	▪ Proactively seeks the views of registrars on their experience ▪ Takes steps to establish a learning community within their department and/or organisation ▪ Monitors, evaluates and takes steps to address areas for improvement in teaching and learning
	Notes on why this rating was given:

Read the criteria above for effective and excellent supervisors and then rate yourself as:

Table

Struggling to be effective

Partly effective

Mostly effective

Fully effective

Excellent

3. Teaching and facilitating learning

Table

The effective supervisor	The excellent supervisor also
▪ Has up-to-date subject knowledge and/or skills ▪ Provides direct guidance on clinical work where appropriate ▪ Has effective supervisory conversational skills ▪ Plans learning and teaching episodes ▪ Uses a range of appropriate teaching interventions in the clinical setting ▪ Facilitates a wide variety of learning opportunities ▪ Helps the registrar develop an ability for self-directed learning ▪ Allows the registrar to make contributions to clinical practice of graduated value and importance commensurate with their competence ▪ Uses technology-enhanced learning where appropriate, e.g. simulation ▪ Encourages access to formal learning opportunities, e.g. academic time	▪ Demonstrates exemplary subject knowledge or skills ▪ Understands and can apply theoretical frameworks to their practice ▪ Is involved with curriculum development beyond the supervisory relationship ▪ Works with the department and/or provider to ensure a wide range of learning opportunities is available, e.g. simulation facilities, courses
	Notes on why this rating was given:

Read the criteria above for effective and excellent supervisors and then rate yourself as:

Table

Struggling to be effective

Partly effective

Mostly effective

Fully effective

Excellent

4. Enhancing learning through assessment

Table

The effective supervisor	The excellent supervisor also
▪ Regularly observes the registrar’s performance and offers feedback ▪ Plans and/or monitors assessment activities ▪ Uses workplace-based assessments appropriately ▪ Provides feedback that is clear, focused and aimed at improving specific aspects of registrar performance ▪ Supports the registrar in preparation for College exams	▪ Understands and can apply theoretical frameworks relevant to assessment to their own and others’ practice ▪ Is involved in professional assessment activities beyond the supervisory relationship, e.g. as College examiner
	Notes on why this rating was given:

Read the criteria above for effective and excellent supervisors and then rate yourself as:

Table

Struggling to be effective

Partly effective

Mostly effective

Fully effective

Excellent

5. Supporting and monitoring educational progress

Table

The effective supervisor	The excellent supervisor also
▪ Agrees an educational contract at the outset of the training period ▪ Understands the curricular requirements of the specialty and stage of training ▪ Identifies with registrar the learning needs and educational objectives ▪ Reviews and monitors progress through regular timetabled meetings ▪ Ensures that appropriate records are kept in relation to registrar progress ▪ Uses the portfolio of learning appropriately and encourages its use by registrars ▪ Provides continuity of supervision or ensures effective educational handover ▪ Responds efficiently and effectively to emerging problems of registrar progress ▪ Is aware of, and can access, available support for the registrar in difficulty ▪ Understands their role and responsibilities within the educational governance structures of their university	▪ Proactively seeks out opportunities for providing formal support and career development activities for registrars ▪ Establishes and/or evaluates schemes for monitoring registrar progress across the department
	Notes on why this rating was given:

Read the criteria above for effective and excellent supervisors and then rate yourself as:

Table

Struggling to be effective

Partly effective

Mostly effective

Fully effective

Excellent

Appendix B: Feedback from registrars

Introduction

Explain the purpose of the interview and that feedback given to the supervisor will be anonymised and given as part of a summary of what has been said.

Establish who is present and what year each registrar is in.

What do you value your trainer for?

Comments:

Table

Are there any areas where you have difficulty gaining experience?

Do you assess your skills regularly in the logbook?

Comments:

Table

Are you able to learn from other members of the healthcare team?

Comments:

Table

How are you involved in constructing your learning plan?

Are learning plans reviewed regularly?

Comments:

Table

Are your consultations/procedures observed regularly and do you get useful feedback?

Comments:

Table

What educational meetings do you have with your on-site supervisor?

Comments:

Table

Do you receive useful assessment and feedback at the end of an allocation/every 6 months?

Comments:

Table

Are you adequately supported in your use of the portfolio?

Comments:

Table

Are you adequately supported in preparing for the FCFP exams?

Comments:

Table

Have you identified any other issues about the educational experience and how were these resolved?

Comments:

Table

Visitor use:

Highlights & recommendations based on the registrar interview:

Table

Appendix C: Observation of clinical training

Background information

Table

Visitors please view and comment where required:

Table

Area		Comment
Evidence of learning outcomes	Y/N
Clarity of expected learning outcomes	Y/N
Evidence of preparation	Y/N
Acceptable quality teaching resources	Y/N
Appropriateness of educational style	Y/N
Appropriate involvement of the registrar	Y/N
Appropriate learner-centredness	Y/N

Visitor’s summary & recommendations:

Table

Appendix D: Developmental goals and planning

Date of visit:

Name of clinical trainer:

Name of visiting family physician:

General comments on highlights and successes or areas that need improvement

Table

Specific goals for the next year

Table

1.

2.

3.

4.

5.

Actions to be taken to achieve these goals

Table

Plan of action	By whom?	By when?
1.
2.
3.
4.
5.

Signature of clinical trainer                  Signature of visiting family physician