Blueprinting for clinical performance deficiencies – Lessons and principles from the General Medical Council's fitness to practise procedures

Abstract

Background: The UK General Medical Council (GMC) in its regulatory capacity conducts formal tests of competence (TOCs) on doctors whose performance is of concern. TOCs are individually tailored to each doctor's specialty and grade.

Aims: To describe the development and implementation of an electronic blueprinting system that supports the delivery of TOCs.

Method: A case study that describes the evolution of the GMC electronic blueprint including the derivation of its content and its functionality.

Results: A question bank has been created with all items classified according to the competencies defined by Good Medical Practice. This database aids test assembly and ensures that each assessment maps across the breadth of the blueprint.

Conclusions: The blueprint described was easy to construct and is easy to use. It reflects the knowledge, skills and behaviours (learning outcomes) to be assessed. It guides commissioning of test material and enables the systematic and faithful sampling of common and important problems. The principles described have potential for wider application to blueprinting in undergraduate or clinical training programmes. Such a blueprint can provide the essential link between a curriculum and its assessment system and ensure that assessment content is stable over time.

Introduction

Dealing with doctors whose fitness to practise (FtP) is called into question is one of the four main functions of the United Kingdom's General Medical Council (GMC) as defined by the Medical Act (GMC 1983).

A system for processing complaints about doctors comprises an initial screening in which all complaints are reviewed and assessed within 1 week; 39% of cases are closed at this stage as the information submitted does not raise a question about the doctor's FtP. For those in whom clinical performance is considered to be the cause of the problem, an FtP performance assessment is initiated (GMC 2004).

An FtP performance assessment comprises two phases: a peer led workplace-based assessment and a series of objective tests of competence (TOC) (Sturrock et al. 2006). The workplace-based assessment includes a site visit, case notes review, case-based discussion, third party interviews and where possible, direct observation of the doctor's practice. The aim of the peer-led workplace assessment is to review a sample of the whole of the doctor's clinical practice in the light of complaints to the GMC and to compare it with the standards expressed in Good Medical Practice (Southgate et al. 2001). The aim of the TOC is to collect evidence about a doctor's knowledge and skills in an examination situation, to allow the GMC FtP panel to make informed decisions on that doctor's FtP.

Unlike other high stakes of medical assessments designed for large volumes of candidates, each GMC TOC is unique. As it is offered to only one doctor at a time, it is tailored to both their specialty and grade using a ‘core and options’ approach. The assessors are guided in the test assembly by the individual doctor's submitted portfolio, thus producing an assessment that accurately reflects both their experience and current work practice. The test content includes knowledge and skills that may be generic for all doctors, specific to the relevant specialty and where appropriate super specialty material.

The TOC consists of a machine-markable written knowledge test and an objective structured clinical examination (OSCE) to assess the doctor's practical and communication skills as well as their clinical method. When designing each test of competence, the content is determined by ‘blueprinting’ the assessment content, in keeping with best educational practice.

In this article, we describe how the electronic blueprinting of the TOC for the GMC's FtP procedure has been developed to fairly and robustly assess doctors referred because of concerns about their clinical performance.

Table

Practice points

Description of the development of an electronic blueprinting system within the context of GMC FtP. Discussion of the general principles of blueprinting. Demonstration of the functionality of the blueprint including approaches to electronic question bank management. Linkage of curriculum with its assessment system and strategies for test construction and sampling.

Blueprinting

Blueprinting involves developing a template used to define the content of a given test. In medical education, it is often designed as a matrix or a series of matrices and can be adapted according to the assessment being developed. In order to produce a matrix that is both fit for purpose and easy to use several key steps need to be completed.

The first step is to define a clear statement of the purpose of the assessment that you are developing (Dauphinee 1994; Schuwirth & van der Vleuten 2006). The GMC TOC is designed to provide a formal test of the basic knowledge and skills required for the area of practice in which the doctor is engaged, e.g. general practice. The next stage is to ensure that each assessment has ‘content validity’, that is, it covers the learning objectives that have been clearly defined. For a GMC TOC, the learning objectives are the standards set out in Good Medical Practice (GMC 2006).

To ensure that each assessment has content validity, the items included in the assessment should be mapped on to a blueprint. (Van der Vleuten et al. 1991; Van der Vleuten 1996) This blueprint is typically a table or spreadsheet which lists contents along the vertical axis and processes or contexts along the horizontal. In the GMC FtP assessment blueprint, the vertical axis contains diseases or disorders and the horizontal axis contains the chapter headings found in Good Medical Practice.

Each assessment should then systematically sample from this blueprint using a test specification or sampling grid (Schuwirth & van der Vleuten 2006). This ensures that each assessment tests the same breadth of knowledge even though different items have been included. The grid can be weighted to reflect the relative importance of the clinical problems likely to be encountered by the candidate. This minimises the possibility of ‘construct under-representation’, that is, the under-sampling or biased sampling in an assessment system (Hamdy 2006). The test specification for each doctor undergoing a TOC is based on the doctor's specialty/subspecialty and grade as detailed in a portfolio.

The blueprint as described is a two-dimensional table, but additional information can be incorporated, either by hyperlinks or as attached Word documents. This enables the titles to remain succinct but able to convey additional information, e.g. grade of doctor that the item is designed to test.

By mapping the pool of all available items on to a blueprint, it is also possible to identify areas in which more items need to be written.

Evolution of the GMC FtP electronic blueprint

The GMC FtP TOC assessment blueprint has been developed using the GMC Professional and Linguistics Assessment Board (PLAB) examination blueprint as a template. Both examinations are based on the principles and values set out in Good Medical Practice (GMC 2001); therefore, the headings of this document are used to determine the horizontal (x) or process axis.

When the original PLAB blueprint was being developed, it was designed to be applicable to all doctors rather than any individual specialty. To determine the vertical (y) headings, PLAB performed workload surveys in accident and emergency (A&E) departments, sent questionnaires to recent candidates and analysed hospital discharge data to determine what doctors who passed the test would be expected to know and to do. In 2004, the original blueprint was revised and redesigned to reflect data about common and important conditions seen by both hospital doctors and general practitioners (Office for National Statistics 1998; NHS Scotland 1999; RCGP 2004; Department of Health 2005, 2007).

When the concept of an FtP blueprint was being discussed, it was noted that as the TOC was designed to provide a formal test of the basic knowledge and skills, the level was equivalent to the PLAB examination. It was therefore felt appropriate to use the same blueprint headings that had been developed for the PLAB examination.

Since the FtP blueprint has been in use, the GMC has published a revised version of Good Medical Practice (GMC 2006). The new document reflects an increased need for working in partnership with patients, there is also increased emphasis on equality and diversity of health care provision and prominence of specific aspects of professional behaviour. The FtP blueprint has been modified to incorporate these changes by making minor changes in the column headings in the areas of professionalism and knowledge of disease factors.

Using the FtP electronic blueprint

All the test materials (questions and cases) used in the GMC FtP process are stored in a question bank designed by Speedwell computer systems. The basic question banking software is available commercially; minor modifications were made to fit our purpose.

Within the question bank, each question is labelled with the question type (e.g. single best answer, extended matching or OSCE station) and the appropriate blueprint category (e.g. women's health, breast lump and diagnosis). It also has keywords such as appropriate specialty/specialties and appropriate grade of doctor. Previous performance statistics are also stored with each item.

On receipt of the individual doctor's portfolio, the appropriate questions can be viewed using a bank overview which shows all the questions available in the appropriate blueprint category. These questions are colour coded according to their level of difficulty when tested on volunteer doctors. Questions can then either be selected individually from the entire bank or appropriate questions can be identified on the basis of their facility index, point biserial coefficient, type of question, appropriate specialty or any combination of these. Alternatively, a sampling grid can be programmed to automatically select a possible paper.

For the ‘core’ component of the assessment, questions with high facility index (0.7–1.0) are selected as this part of the assessment is designed to test basic knowledge. The ‘option’ part of the assessment is then tailored to the individual doctor in terms of area of practice and grade of doctor, e.g. for a consultant gastroenterologist, and questions with lower facility index but positive point biserial coefficients (i.e. discriminates well) within the digestive area of the blueprint are selected.

After selecting the questions that are to be included in the assessment, using the Speedwell software, an examination overview can be automatically created showing the position of each question in relation to the FtP blueprint (Figure 1). Using this overview, each examination is checked to ensure that there is an appropriate range of questions in terms of subject content (the y-axis of the blueprint) and also that all areas of Good Medical Practice (GMC 2006) (the x-axis) are represented.

Figure 1. The initial rows of an examination blueprint for general medicine single best answer paper.

Discussion

Blueprinting is an integral component in the design of any assessment system. To be effective, blueprints should:

• be easy to use

• be designed to reflect the length and breadth of the curriculum that is being assessed

• guide the test designers with regard to commissioning item/case writers

• provide the essential link between the curriculum and its assessment system

• enable systematic and faithful sampling of common and important problems using the derived test specification, so ensuring the assessment content is consistent over time.

We have described the development of an electronic blueprint that is being used to compile tailored TOCs as part of the GMC's FtP procedures. As each doctor in these procedures undergoes an individualised test based on his or her actual practise, it was necessary to automate the blueprinting process to allow rapid and flexible selection of an appropriate test of knowledge set at an appropriate level. The software functionality has been developed specifically for this purpose. The automation of the blueprinting process allows more flexible and appropriate selection of assessment questions for individual doctors. As such, it should be a suitable tool to support the selection of the more specialty-based and specific assessment requirements likely to be needed for revalidation.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

Additional information

Notes on contributors

David Sales

DAVID SALES is a general practitioner and consultant in medical education.

Alison Sturrock

ALISON STURROCK is a senior clinical lecturer in medical education and an honorary consultant physician.

Katharine Boursicot

KATHARINE BOURSICOT is a reader in medical education.

Jane Dacre

JANE DACRE is a professor of medical education and vice dean and director of education at University College London.