Local and national effects of a quality system in Dutch general practitioner specialty training: a qualitative study

Abstract

A quality system (named GEAR; acronym for Combined Evaluation Audit Round in English), has been introduced in eight institutes of the Dutch general practitioner specialty training. This paper focuses on the local and national effects of GEAR. Seventeen semi-structured interviews were conducted with the directors and quality co-ordinators. At a local level, GEAR provided the institutes with insights into their current practice. The institutes designed and implemented several improvement plans based on feedback. Furthermore, GEAR addressed quality management more systematically, and it enhanced the quality culture. At a national level, the institutes formulated national priorities. In addition, GEAR stimulated collaboration as a result of more frequent contacts. Institutes differed in their perception of the extent to which GEAR offered added value. Integrating the quality system into the local policy was sometimes a problem. Despite some scepticism at the start, GEAR provoked enthusiasm that can contribute to the quality of medical education.

Keywords:

• Internal quality assurance
• collaboration
• medical education
• quality system

Introduction

Research into the effects of internal quality systems (QSs), working according to quality principles, show positive effects at the organisational level (Doeleman et al., 2013; Kleijnen et al., 2011): QSs can lead to more structured processes, they can lead to discussion of points for improvement, which can then be addressed, and they can lead to more coherence within an organisation because they involve all stakeholders (Da Dalt et al., 2010). QSs may further lead to more attention to educational methods and to the students’ perspective (Brennan & Shah, 2000). Internal QSs tend to focus more on quality enhancement than on assurance (Becket & Brookes, 2006). It would be helpful to gain a deeper understanding and more insight into the factors making institutes benefit from working with a QS (Kleijnen et al., 2011). Qualitative research may play a part in gaining more insight and in establishing if and how the benefits were achieved (Cambon et al., 2012). This qualitative study explored whether a QS, implemented in the Dutch general practitioner specialty training, is conducive to quality assurance and improvement both at the institute level and the national level and whether the system stimulates mutual cooperation.

Quality assurance in medical education

As the quality of medical education affects the healthcare, accreditation systems are increasingly considered relevant (van Zanten et al., 2008). Such accreditation systems are commissioned by an authority that assesses educational institutes by clear standards (van Zanten et al., 2008; Hillen, 2010), and their ultimate aim is to improve the quality of healthcare (van Zanten et al., 2008; Davis & Ringsted, 2006; Hillen, 2010). The standards themselves require periodic assessment, for example with self-evaluations and peer-reviews, or a combination of these methods. Together, the World Health Organisation and the World Federation for Medical Education have developed standards for medical education, which can be used internationally and are already used in many countries for accreditation purposes (Karle, 2006). These standards encompass aspects such as structure, content, processes, management and assessment (Karle, 2006; Christensen et al., 2007). Accreditation programmes are mostly developed at a national level to assure, and often also to improve, the quality of medical education at a local level. In addition to being subjected to national accreditation, institutes increasingly develop their own QSs; define their own standards, and choose methods to assess their standards for internal use (Brennan & Shah, 2000; Buwalda et al., 2017a).

The Dutch general practitioner specialty training

The eight Dutch general practitioner specialty training institutes decided to cooperatively develop a national QS. All university hospitals in the Netherlands have their own general practitioner specialty training institute. The institutes share the educational attainment of the training but each institute is responsible for its own educational programme and management (Figure 1).

Figure 1. General practitioner specialty training in the Netherlands.

Quality system ‘GEAR’

In order to further improve the quality of the training institutes, the eight Dutch general practitioner specialty training institutes wanted to create a structured internal QS with clear and shared criteria, also suitable for benchmarking. The main goal of such a new QS would be to support the general practitioner specialty training institutes in making plans for improvement, implementing these plans, and exchanging products. Consequently, the QS should stimulate the institutes to improve their own quality and contribute to the quality of the other institutes. Thus, GEAR (Dutch acronym for Combined Evaluation Audit Round) is an internal QS that uniquely focusses on quality assurance and improvement on both the local level (institutes) and on the national level. The differences between accreditation, internal QSs and GEAR are shown in Table 1.

Table 1. Differences between accreditation and GEAR.

Accreditation	Internal QS	GEAR
Developed at a national level	Developed at a local level	Institutes cooperatively developed GEAR (local and national level)
Focusses on quality assurance and improvement at a local level	Focusses on quality assurance and improvement at a local level	Focusses on quality assurance and improvement at a local and national level
Tend to focus more on assurance	Tend to focus more on quality enhancement	Tend to focus more on quality enhancement

GEAR has seven domains (Figure 2) that cover the broad elements in the structure, process and outcome of the general practitioner specialty training: (1) staff; (2) management; (3) vision and quality policy; (4) academic level; (5) assessment, evaluation and results; (6) educational environment; and (7) educational programme. For each domain indicators were formulated that are assessed by means of self-evaluation and an audit (qualitative and quantitative feedback). An important aspect of the system is that it facilitates the exchange of ‘good practices’ such as educational interventions, local policies and ideas to approach organisational problems. This is accomplished by the provision of benchmark results in order to detect areas of excellence at other institutes and by organising exchange meetings with the quality co-ordinators of the eight institutes. Furthermore, as a part of GEAR, a national quality co-ordinator was appointed to stimulate the exchange and collaboration between institutes. The national quality co-ordinator is also tasked with the continuous development of the QS based on evaluation results and new insights. This to assure the QS will be up-to-date and relevant to the current quality issues at the institutes.

Figure 2. Quality System GEAR. Source: Adapted from Buwalda et al. (2017a) and Buwalda et al. (2017b).

Development and implementation

GEAR has been set up systematically in small steps while engaging with the target groups. The directors of the general practitioner specialty training institutes were closely involved in the development of GEAR and they were tasked with the introduction and implementation of the system at the institutes. Not only directors were involved but also clinical trainers, trainees, experts in quality management and representatives of professional associations, including a representative from the national controlling committee. The development and implementation of GEAR is detailed in previous studies (Buwalda et al., 2017a; Buwalda et al., 2017b). It appeared that formulating goals together with stakeholders at the start was an important part of the process: this gave direction concerning the design of the system. Furthermore, the collaboration encouraged the support of, and commitment from, stakeholders regarding the QS.

At all institutes quality co-ordinators are available to coordinate the data collection (coming from the self-evaluation) and they support the designing and implementation of the improvement plans that arise from GEAR. In the previous implementation study (Buwalda et al., 2017b) it turned out that, most quality co-ordinators felt the QS was imposed on them. This led to the recommendation to also involve the quality co-ordinators in the development and implementation of a QS. Although they were mainly extrinsically motivated to participate; they did introduce, implement and use the QS. Peer pressure and a sense of togetherness among the quality co-ordinators was an important stimulator for them. This confirms previous findings that relatedness can contribute to an effective implementation: working together can enhance confidence and motivation, and prevent isolation (Knowles, 1975; Damschroder et al., 2009; Dixon-Woods et al., 2012; Rafferty et al., 2012; Cook & Artino, 2016).

Methods and data

Between May and August 2015, almost three years after GEAR was introduced, all the directors (quotes indicated by D below), quality co-ordinators (QC) and the national quality co-ordinator were interviewed individually to explore the effects of GEAR (n = 17); and all participants gave informed consent. A week before the planned interview, participants received an email with the topic list to prepare. The researcher who interviewed all participants (NB) received the improvement plans of all institutes prior to the interview, to prepare (an example is given in Figure 3).

Figure 3. Example of an improvement plan.

The topic-list was based on the three aims of GEAR: quality assurance, quality improvement and collaboration. Potential local and national effects for each of these aims were listed beforehand and could be checked during the interviews. On average, the interviews lasted one hour.

The main researcher (NB) undertook a content analysis of the interviews (Pope et al., 2000). All interviews were audio-recorded and transcribed verbatim. The process started with open coding from the transcripts. NB distinguished important fragments and developed a tree diagram to organise the resulting categories. The tree diagram consisted of two main categories: local effects and national effects; and these could be divided in subcategories (awareness, involvement, culture, structure, products and insights). For example, if a participant mentioned he or she did or did not gain new insights, this fragment was categorised as ‘local effect’ and subheading ‘insight’. To prevent the narrowing of ideas, the tree diagram was discussed by two researchers (NB and MV) after coding one, three and six interviews. After coding all the interviews, NB reread the interviews to evaluate completeness. Then, the categories could be structured in four main themes, which was discussed within the entire research team,

Results: observed effects of GEAR

The interviews showed four central themes: providing performance insight, systematically working according to quality principles, enhancement of the quality culture and collaboration, which will be described separately.

GEAR as a tool to provide performance insight

All participants agreed that GEAR gave insight into the performance of the institute on the measured domains and it allowed them to compare their own performance to the other institutes (benchmarking). Four institutes explicitly mentioned that they were able to identify new insights. For example, one institute found out that it did not pay attention to excellent trainees. Therefore, they developed an improvement plan with the aim of offering a special programme for such trainees.

Some also mentioned that the GEAR feedback confirmed their weak point:

GEAR has produced much information. Although we already knew our weak point, owing to the outcomes of the measurement round, we again discussed our weak point with the aim to improve it. (QC2)

Most GEAR feedback was familiar to the institutes. However, a participant disagreed with their scores on one domain and therefore found the benchmarking on that point worthless.

All institutes developed improvement plans based on the feedback coming from GEAR.

At a national level, GEAR showed that the institutes had shared concerns regarding the subjects academic level and assessment. Knowing these two common concerns of all institutes, the national quality co-ordinator developed two improvement plans commissioned by the directors of the general practitioner training institutes. Only a few participants were aware of these plans. Most of the directors felt GEAR should pay more attention to common difficulties: ‘I would like GEAR to inform us more and point out: “these are the common difficulties”. We could then examine them closely’ (D3).

GEAR as a tool to work systematically according to quality principles

Following the steps of GEAR (Figure 1) provided the institutes with a firm structure to work from according to quality principles. Most participants shared this view. Since such a firm structure was lacking at their own institute, it proved to be helpful for most institutes. Two institutes did already apply similar principles and they benefitted less from the QS structure:

Before GEAR was introduced we had developed our own quality management system. It proved difficult to integrate GEAR. We wanted to use our own system, and to explore how we could use GEAR. (D2)

Participants mentioned that the improvement plans helped them to describe the current and desired situation and the resulting targets (see Figure 2 for an example). Moreover, GEAR supported a systematic implementation of the planned improvements, in which one of the staff members was responsible for the implementation and systematic evaluation of the plan(s). This was helpful in finalising the plan.

Participants mentioned that, as an effect of the introduction of GEAR, all the institutes had reserved time and staff for improvement plans. For example, before the introduction of GEAR, the majority of the institutes did not have a quality co-ordinator but now appointed one who was responsible for quality work at the institute: ‘I give structure to quality work at our institute, and I monitor the progress of the improvement plans’ (QC6).

At a national level, GEAR led to regular and structured quality co-ordinator meetings. Once a month, all directors met to discuss important national issues, and GEAR became a recurring item at those meetings. The directors also discussed the outcomes and the further development of GEAR in these meetings.

GEAR as a tool to enhance the quality culture

The quality co-ordinators in particular, but also most of the directors, mentioned that their attitude to working with the QS changed. They became increasingly aware that GEAR was an incentive for improvement at their own institute. In the beginning, participants were more focused on the obligation to participate but gradually they began to understand and appreciate GEAR. Understanding the benefits of GEAR made participants feel ready for the next measurement round; and one participant mentioned that she was actually looking forward to it. In addition to the changing attitude towards GEAR, participants indicated that more people were involved in the improvement projects at the institutes. For example, during meetings they discussed the improvement plans, talked about implementation and how to proceed.

… putting it on the agenda, talking about it, involving everyone in it. Yes, when plans are being implemented, everyone should act seriously. Keep the process alive. Because the directors and the managers consider the plans vitally important, everyone takes them seriously. (QC5)

At a national level, participants believed that the transparency between institutes had grown because the institutes showed each other their strengths and weaknesses. A few participants mentioned that it was now easier than in the past to show weaknesses to each other. At GEAR meetings, the participants discussed quality work and identified difficulties and solutions. The participants reported that the regular attention resulted in more contact between institutes than before the introduction of GEAR. It was mentioned that before the introduction of GEAR, the quality co-ordinators mostly worked on their own institute, focusing on their local policy only:

Because of GEAR, we now discuss quality and meet each other. Before, we all worked on quality, but without contacting other institutes. These days I do, and that is one of the benefits of GEAR. (QC7)

…Before this QS, we operated entirely on our own. (D2).

GEAR as a tool to stimulate collaboration

Most participants agreed that there was more cooperation between institutes. The quality co-ordinators regularly met and exchanged good practices. During these co-ordinator meetings, they presented products and also discussed how to implement improvement plans or how to involve others in quality work, for example.

Regular meetings with all the QCs and talking about quality is a benefit of GEAR. At these meetings, we learn and we can support each other. (QC8)

Most quality co-ordinators mentioned that they had come to know each other better during the introduction of GEAR. This enabled them to contact each other if they had questions; consequently, they got in touch with other co-ordinators more often than before.

The exchange between institutes led to the introduction of products and ideas at their own institute. The extent to which institutes benefitted from this differed. A few institutes introduced concrete products and ideas from other institutes. Other institutes, however, did not. They concluded that using products of other institutes was complicated because of the different structures and formats of the institutes.

Conclusions

The purpose of the research reported in this article was to explore the effects of GEAR in eight Dutch general practitioner specialty training institutes. The aim of this common quality system is to stimulate quality assurance, improvement and collaboration between institutes. This study focused on the extent to which these goals were achieved at a local and national level conducting seventeen semi-structured interviews with the directors and quality co-ordinators. The results indicate four major advantages of using GEAR: providing performance insight; systematically working according to quality principles; enhancing the quality culture; and stimulating collaboration. The results also revealed some difficulties as recommendations of a common QS can sometimes impinge on existing, local routines.

The insights gained by GEAR encouraged most institutes to develop and implement improvement plans, which stimulated quality assurance and quality improvement at the local level. The clear structure of GEAR and the ensuing improvement plans caused institutes to take steps to improve quality. It also created more cooperation among staff. These results further confirm previous findings that QSs may lead institutes to create more structure in quality work, tackle points for improvement and their staff are more involved in making decisions concerning quality improvement (Da Dalt et al., 2010). In addition, the results indicate that the institutes made resources (time and staff) available to work on quality, that attitudes changed and that institutes introduced products and ideas from other institutes.

The institutes benefitted from the QS in different ways, which corroborates the results of previous studies (Brennan & Shah, 2000; Kleijnen et al., 2011; Kleijnen et al., 2014), although, some institutes questioned the balance between investments and benefits, because, for example, the feedback offered little new insight into familiar problems. Some of these institutes already had an internal quality system and did not explicitly need GEAR to structure their quality work. Furthermore, the introduction of good practices provided by other institutes did not always go down well mainly because it did not fit the local situation. The added value of GEAR in these situations was more at the national level: common weaknesses of all institutes formed the basis of national improvement plans; directors and quality co-ordinators regularly discussed the QS; and there was more transparency and collaboration among the institutes. This study shows the importance of this contact between institutes.

With previous findings and theories (Knowles, 1975; Damschroder et al., 2009; Dixon-Woods et al., 2012; Cook & Artino, 2016) in mind, this collaboration seems to be very important. It could explain why attitudes changed from more extrinsically motivated to intrinsically motivated; during the implementation process, especially the quality co-ordinators were more focused on the obligation to participate; but this study reveals that gradually they began to understand and appreciate the QS. Institutes became aware that the system was an incentive for improvement at their own institute. Talking and thinking about quality both nationwide and at individual institutes was experienced as positive and stimulating. Institutes started to realise that working on quality was not because institutes had to meet criteria but because they want to deliver high-quality education. This mind-set, that quality is ‘value from inside out’ is important to establish working with QSs (Guaspari, 1987; Leebov and Ersozs, 2003). The directors and quality co-ordinators at the institutes play an important role in taking leadership in the pursuit of high-quality education and to carry this mind-set to get all staff involved (Guaspari, 1987; Lomas, 2004). Introducing a collaborative QS at the Dutch general practitioner specialty training institutes encouraged them to think and talk about quality and they gradually began to apply quality principles and working on quality became a more natural process.

There are some limitations to this study. First, it is not certain that the effects of this study are due only to GEAR. Other factors may have played a role as well, such as the increasing recognition of QSs in general and the recognition of the institutes to collaborate and reach uniformity at a national level can be an effect and also be a cause of GEAR. It is difficult to isolate the effects because national and local policies also influence institutes (Harvey, 2006). However, in the interviews the improvement plans were explored and collaboration activities related to GEAR only, which suggests that the answers that were processed pertain to the direct effects of the introduction of GEAR. A second limitation is that it is not known to what extent the reported effects will actually lead to the improvement of the training institutes or their education and whether they will deliver better general practitioners. However, it is known that audit and feedback systems improve quality of care (Ivers et al., 2012).

A strength of the study is that all the professionals of eight institutes that were directly involved with the introduction of GEAR were interviewed. The same topic list was used for all participants and the systematic analyses made it possible to gain a clear overview of all the perspectives.

The findings of this study indicate that a quality system such as GEAR, can certainly contribute to the quality of higher education. However, more research is needed into how institutes can learn and benefit more from each other and how QSs affect education and its contribution to a better practice. Future research could build on these results, the themes (providing performance insight; systematically working according to quality principles; enhancement of the quality culture; and collaboration) could be used as a framework and could be interesting building blocks for a questionnaire construction.

Disclosure statement

Dr M.R.M. Visser also works at the Dutch General Practitioner specialty training institute as a quality co-ordinator and is member of the national team tasked with the development and maintenance of GEAR.

Ethics approval and consent to participate

Participants gave informed consent.

Funding

This work was supported by the Stichting BeroepsOpleiding Huisartsen (SBOH).

Acknowledgements

The authors are sincerely grateful to Jeroen Zaat for his assistance in organising this study, and to Simon Muskitta for the design of the GEAR figure. Furthermore, we would like to thank all Dutch general practitioner training institutes and the national quality co-ordinator for their hospitality and their co-operation.