The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis

Abstract

Background: Assessment of clinical teaching by learners is of value to teachers, department heads, and program directors, and must be comprehensive and feasible.Aims: To review published evaluation instruments with psychometric evaluations and to develop and psychometrically evaluate an instrument for assessing clinical teaching with linkages to the CanMEDS roles.Method: We developed a 19-item questionnaire to reflect 10 domains relevant to teaching and the CanMEDS roles. A total of 317 medical learners assessed 170 instructors. Fourteen (4.4 %) clinical clerks, 229 (72.3%) residents, and 53 (16.7%) fellows assessed 170 instructors. Twenty-one (6.6%) did not specify their position.Results: A mean number of eight raters assessed each instructor. The internal consistency reliability of the 19-item instrument was Cronbach's α = 0.95. The generalizability coefficient (Ep²) analysis indicated that the raters achieved Ep² of 0.95. The factor analysis showed three factors that accounted for 67.97% of the total variance. The three factors together, with the variance accounted for and their internal consistency reliability, are teaching skills (variance = 53.25s%; Cronbach's α = 0.92), Patient interaction (variance = 8.56%; Cronbach's α = 0.91), and professionalism (variance = 6.16%; Cronbach's α = 0.86). The three factors are intercorrelated (correlations = 0.48, 0.58, 0.46; p < 0.01).Conclusion: It is feasible to assess clinical teaching with the 19-item instrument that has demonstrated evidence of both validity and reliability.

Introduction

The challenge of how to systematically evaluate clinical teaching has been the subject of much discussion among educators, researchers, and administrators in medical schools concerned with the quality of clinical teaching. Many evaluation instruments have been developed to provide feedback to clinical teachers, department heads, and program directors. There has been a considerable effort devoted to investigate the psychometric properties of instruments to assess clinical teaching effectiveness (Ramsbottom-Lucier et al. 1994; Solomon et al. 1997; Bardes et al. 1998; Litzelman et al. 1998, 1999; Marriott & Litzelman 1998; Copeland & Hewson 1999; Steiner et al. 2000; Williams et al. 2001, 2002; Snell et al. 2002; van der Hem-Stokroos et al. 2003; Beckman et al. 2004, 2005; Leamon & Fields 2005; Bierer & Hull 2007).

Specialty Medical Education in Canada has undergone a significant change since the adoption of the CanMEDS 2000 Project, first published in 1996 as the Skills for the New Millennium Report of the Societal Needs Working Group. Although the CanMEDS competencies are now accepted among educators involved with residency program administration, they are still not clearly understood by many teachers in clinical medicine (Frank 2005).

The evaluation of clinical teaching is a much studied topic in medical education as the results may have significant implications for both clinical teachers as well as program administrators. Teachers, learners, program administrators, and the health care system, as well as patients, can be expected to gain from improved clinical teaching (Snell et al. 2002). Thus, its evaluation is interesting and important.

A common type of teacher evaluation is the administration of a standardized form to rate teachers by learners at the conclusion of clinical rotations. Over the years, medical educators have reported on the development of these forms (Irby & Rakestraw 1981; Litzelman et al. 1998; Copeland & Hewson 1999; Copeland & Hewson 2000; Beckman et al. 2004). Snell et al. (2002) reviewed the research on the evaluation of clinical teaching and outlined a set of guiding principles to broaden current perspectives on evaluation: validity, reliability, efficiency, and feasibility. They also indicated that evaluation instruments should be consistent with the culture of the organization, acceptable to teachers, easy to administer, and applicable to all levels of teachers.

Drawing from previous research (Irby & Rakestraw 1981; Irby 1983; Guyatt et al. 1993; Cohen et al. 1996; James & Osborne 1999; Copeland & Hewson 2000) and relevant articles on effective clinical teaching (Parsell & Bligh 2001; Stark 2003; Sutkin et al. 2008; Pratt et al. 2009), our objective in this study was to review existing clinical instruments and then develop and test a new clinical teaching assessment instrument (CTAI) that could be applied broadly across all clinical departments to assess clinical teaching contributions, provide feedback to clinical teachers, their department heads, and program directors.

The pilot study applied certain conditions to the development of the instrument which included: (1) the instrument must be comprehensive, with evidence of validity and reliability; (2) the instrument must be feasible; that is, it had to be workable, practical, and achievable and (3) the instrument must link to the CanMEDS roles.

Table

Practice points

This article presents a practical 19-item questionnaire that can be used by learners to provide feedback to teachers on their clinical teaching. It can be completed in 5 min and the behaviors assessed are linked to the CanMEDS roles. This article demonstrates that the questionnaire is psychometrically sound with evidence of validity and reliability. The questionnaire therefore may be useful to all teachers who wish to assess their teaching.

Methods

Our first step was to identify and review existing clinical education evaluation instruments. We identified 12 published studies of teaching assessment instruments with associated research on reliability and validity. These are summarized in Table 1.

Table 1.  Summary of psychometric characteristics of 12 clinical teaching assessment instruments.

Authors	Instrument (name, type items, number of items)	Sample type and size	Reliability evidence	Validity evidence
Copeland and Hewson (1999)	Clinical Teaching Effectiveness Instrument (CTEI); five-point scale, 15 questions	711 Clinician-educators	g^a coefficient  =  0.94	Content- and criterion-related validity
Leamon and Fields (2005)	Brief Instructor Rating Scale (BIRS); five-point scale, 11 items	27 Lecturers 276 Students	g^a coefficient range: 0.66–0.95	Principal-component analysis with varimax rotation
Litzelman et al. (1998)	Stanford Faculty Development Program Clinical Teaching Framework (SFDP-58);five-point scale, 58 items	178 Teachers 1581 Students	Cronbach's α range: 0.82–0.95	Factor analysis using principal-component solutions with an oblique rotation for correlated categories on a split sample
Litzelman et al. (1999)	Stanford Faculty Development Program Clinical Teaching Framework (SFDP-26); five-point scale, 26 items	44 Clinical teachers 45 Residents	Cronbach's α  =  0.98	Factor analysis using principal-component solutions with an oblique rotation for correlated categories
Steiner et al. (2000)	Emergency Rotation Scale (ERS); five-point scale, four domains	29 Preceptors 18 Residents	Cronbach's α  =  0.85	Multitrait multimethod
Williams et al. (2001, 2002)	Global Rating Scale (GRS); five-point scale, single item	98 Residents	SEM < 0.05	Criterion-related validity (correlations)
Irby and Rakestraw (1981)	Clinical Teaching Assessment Form (CTAF); five-point scale, nine items	230 Faculty members and residents 320 Students	Corrected Spearman Brown mean  =  0.82	Factor analysis resulted in expected four factors; repeated measures analyses
Cohen et al. (1996)	Teaching Effectiveness of Surgeons (TES); five-point scale, 28 items	43 Surgeons over a 9-year period; 3750 assessments	ICC^b  =  0.65	Scores of good, average, and poor teachers were stable over 9 years
Guyatt et al. (1993)	Evaluating Clinical Teachers in Internal Medicine; 14 domains	41 Physicians	None reported	Factor analysis resulted in one (factor) domain
James and Osborne (1999)	Medical Instructional Quality (MedIQ); six-point scale, 25 items	156 Observations from 131 students	Cronbach's α  =  0.94	Factor analysis with varimax rotation Criterion validity
Magill et al. (1986)	Evaluating Teaching in an Ambulatory Setting	19 Physicians 24 Residents	Interrater agreement on categories >90%	Face and content
McLeod et al. (1993)	Clinical Tutor Evaluation Questionnaire (CTEQ); six-point scale, 25 items	On average: 15 tutors 50 students	Interrater agreement on rankings	Factor analysis using principal-component solutions with varimax rotation

Notes: SEM, standard error of measurement.

^ag, generalizability coefficient; ^bICC, interclass correlation.

While some of the instruments had adequate reliability, each had limitations when considered for widespread use either due to a narrow focus (specific to one medical discipline) or to lack of feasibility due to the time required to complete the instrument. One of our goals was to create an instrument that was useful for all postgraduate medical specialty programs as well as family medicine and can be completed in less than 5 min.

As can be seen in Table 1, we carefully summarized relevant research issues of each study (i.e., sample type and size, instrument item type) and psychometric characteristics (i.e., reliability, validity). While most instruments had adequate internal consistency reliability, few studies reported generalizability coefficients or validity data beyond factor analysis. A notable exception was the Steiner et al. (2000) study which employed a multitrait, multimethod approach to construct validity.

The Beckman et al. (2004) review of the published data on the reliability of instruments points to a lack of articles that focus on examining the psychometric characteristics of scales used to assess clinical teachers. Like our summary in Table 1, they found that while factor analysis and Cronbach's α were the most commonly reported statistical methods used, there was little evidence to support the use of other less common methods that used test–retest reliability, interrater reliability, and convergent validity between instruments.

While Beckman et al. (2004) also summarized themes that could help to develop a general evaluation instrument, they argued that the unique teaching cultures at most schools limit the generalizability of even the most carefully designed instruments and that future studies should focus on more narrowly defined populations.

In a subsequent work, Beckman et al. (2005) emphasized the need to use consistent validity criteria for teaching assessment instruments. They noted, as we have, that most validity studies have focused on content and internal structure evidence (e.g., factor analysis) but less on criterion-related and construct validity (e.g., multitrait multimethod).

Instrument development

After reviewing the instruments detailed in Table 1, many of which had items in common, we considered the CanMEDS (1996) framework as well as The College of Family Physicians of Canada Standards for Accreditation of Residency Training Programs (2006),and designed a 19-item questionnaire to reflect 10 teacher domains relevant to the CanMEDS framework on a five-point scale (1 = never to 5 = always; Table 2). The 10 domains included clear/organized, enthusiastic/stimulating, establishes rapport with learners, actively involves learners, knowledgeable/analytical, provides direction and feedback, approachable, respect/professional behavior, communication, and health advocacy. The CanMEDS roles of manager, communicator, professional, medical expert, scholar, collaborator and health advocate were linked to each question. The item linkages to the CanMEDS roles are shown in Table 2. We called this instrument the CTAI.

Table 2.  Descriptive statistics for the 19 items of the clinical teaching assessment instrument and CanMEDS roles (n  =  317).

	Minimum	Maximum	Mean	SD	NA	CanMEDS roles
1. Was well prepared for teaching sessions^a	2	5	4.24	0.74	52 (16.4)	Manager
2. Organized time to allow for teaching and care giving	2	5	4.13	0.91	14 (4.4)	Manager
3. Was stimulating	2	5	4.25	0.84	2 (0.6)	Communicator
4. Fostered an environment of respect in which I felt comfortable participating	1	5	4.50	0.79	0.0	Professional
5. Coached me on my clinical reasoning or technical skills	1	5	4.11	0.90	9 (2.8)	Medical expert
6. Encouraged me to ask questions	1	5	4.28	0.82	1 (0.3)	Communicator
7. Incorporated research data or practice guidelines into teaching	1	5	4.09	0.94	6 (1.9)	Scholar
8. Emphasized a problem-solving approach rather than solutions	1	5	4.15	0.82	2 (0.6)	Scholar
9. Stimulated me to learn independently	1	5	4.24	0.79	2 (0.6)	Scholar
10. Clearly specified what I was expected to know and do during this rotation	1	5	3.80	1.07	16 (5.0)	Medical expert
11. Offered feedback	1	5	4.00	0.92	5 (1.6)	Scholar
12. Was approachable for discussion	1	5	4.54	0.76	0.0	Communicator
13. Treated team members in a professional manner	1	5	4.58	0.72	0.0	Collaborator
14. Demonstrated compassionate patient-centered care	1	5	4.52	0.70	7 (2.2)	Medical expert
15. Interacted effectively with patients and their families	1	5	4.52	0.69	11 (3.5)	Professional
16. Answered questions clearly	1	5	4.41	0.73	0.0	Communicator
17. Taught effective patient/family communication skills	1	5	4.14	0.92	19 (6.0)	Communicator
18. Pointed out opportunities for health advocacy	1	5	3.85	1.03	25 (7.9)	Health advocate
19. Responded to individual patient health needs as part of patient care	1	5	4.29	0.76	11 (3.5)	Health advocate

Notes: Cronbach's α of coefficient =0.95; mean standard error of measurement =0.19; ^aresponse is on a five-point scale (1  =  never to 5  =  always).

Table 3.  Principal components rotated to the varimax criterion with Kaiser normalization.

Items	Teaching skills	Patient Interaction	Professionalism
1. Was well prepared for teaching sessions	[0.702]	0.228	0.156
2. Organized time to allow for teaching and care giving	[0.753]	0.278	0.258
3. Was stimulating	[0.650]	0.295	0.369
4. Fostered an environment of respect in which I felt comfortable participating	0.370	0.212	[0.763]
5. Coached me on my clinical reasoning or technical skills	[0.629]	0.404	0.250
6. Encouraged me to ask questions	[0.658]	0.115	0.496
7. Incorporated research data or practice guidelines into teaching	[0.734]	0.169	0.183
8. Emphasized a problem-solving approach rather than solutions	[0.583]	0.387	0.341
9. Stimulated me to learn independently	[0.634]	0.182	0.395
10. Clearly specified what I was expected to know and do during this rotation	[0.744]	0.264	0.113
11. Offered feedback	[0.593]	0.361	0.131
12. Was approachable for discussion	0.441	0.177	[0.705]
13. Treated team members in a professional manner	0.162	0.348	[0.787]
14. Demonstrated compassionate patient-centered care	0.173	[0.731]	0.454
15. Interacted effectively with patients and their families	0.182	[0.766]	0.398
16. Answered questions clearly	0.280	[0.684]	0.387
17. Taught effective patient/family communication skills	0.284	[0.832]	0.132
18. Pointed out opportunities for health advocacy	0.394	[0.730]	−0.046
19. Responded to individual patient health needs as part of patient care	0.349	[0.775]	0.195
Percent of variance	53.25%	8.56%	6.16%
Cronbach's α	0.92	0.91	0.86
Correlations between factors
Teaching skills	–
Patient interaction	0.48**	–
Professionalism	0.58	0.46	–

Note: Rotation converged in seven iterations; **p  <  0.01.

Procedures

All postgraduate medical education programs in the Faculty of Medicine at the University of Calgary were invited to participate. Participating departments included Adult Cardiology, Adult Hematology, Adult Infectious Diseases, Adult Respirology, Anesthesia, Community Medicine, Emergency Medicine – CCFP, Emergency Medicine – FRCP, Family Medicine, Internal Medicine, Obstetrics and Gynecology, Pediatrics, Pediatrics subspecialities, Radiation Oncology, Radiology, Surgery, and Surgery subspecialities. Each clinical program director assisted in administering the instrument and was asked to provide the questionnaires at an educational activity. Participants were assured of the anonymity of their responses which were collected in confidential sealed envelopes and delivered for data analysis.

Analyses

A number of statistical analyses were undertaken to address the reliability and validity of the instrument. Response rates were used to determine feasibility for each item on the CTAI, therefore the percentage of “not applicable” along with the mean and standard deviation was computed. We propose that items where “not applicable” exceeds 20% on an instrument may be in need of revision or deletion as learners do not see them as evaluable behaviors.

Instrument internal consistency reliability was examined by calculating the Cronbach's α coefficient. This analysis was followed by a generalizability analysis (G-study) for a single-facet, nested design to determine the generalizability coefficient (Ep²) to insure there were sufficient numbers of items and raters to provide stable data for each individual instructor. Normally, an Ep² ≥ 0.70 suggests data are stable. If the Ep² is low, it suggests that more raters or more items would be required to enhance stability.

We used exploratory factor analysis to determine which items belonged together (i.e., became a “factor” or “scale”). This analysis allowed us to identify the factors, and describe the relative variance accounted for by each factor and their coherence. Using individual rater data as the unit of analysis, the items were intercorrelated using Pearson product moment correlations. The correlation matrix was then decomposed into principal components and these were subsequently rotated to the normalized varimax criterion. Items were considered to be part of a factor if their primary loading was on that factor. The number of factors to be extracted was based on the Kaiser rule (i.e., eigenvalues >1.0).

The factors or scales established through exploratory factor analysis were used to establish the key domains (e.g., CanMEDS roles), while the items within each factor provide more precise information about specific behaviors (e.g., organized time to allow for teaching and care giving, interacted effectively with patients and their families). This analysis makes it possible to determine whether the instrument items were aligned into the appropriate constructs, CanMEDS roles or factors as intended.

This study received approval from the Conjoint Health Research Ethics Board of the University of Calgary.

Results

A total of 317 clinical clerks, residents, and fellows assessed 170 instructors. Of these, 14 (4.4%) were clinical clerks, 229 (72.3%) were residents, 53 (16.7%) were fellows, and 21 (6.6%) did not specify their position. When asked their degree of involvement with the clinical teacher/resident during the current rotation, 28 (8.8%) reported a slight involvement, 84 (26.5%) reported having a moderate involvement, 113 (35.7%) reported a considerable involvement, 73 (23%) reported having extensive involvement, and 19 (6%) did not report on their degree of involvement. A mean number of eight raters assessed each instructor. When presented to postgraduate medical education program directors for acceptability, 18 program directors from a variety of medical and surgical specialties and subspecialties including Family Medicine found the instrument acceptable and volunteered to allow their residents to participate in this pilot study.

The analyses for the CTAI are reported under two headings.

Item analyses, reliability, and generalizability

The item analyses are summarized in Table 2. The range of item means for all 19 items were 3.80 (SD = 1.07) to 4.58 (SD = 0.72). The overall mean was 4.24 and SD = 0.83 of one full-scale point. The distributional properties of the items on the CTAI are typical of five-point items. No items on the CTAI had levels of “not applicable” greater than 20%, though one item (Was well prepared for teaching sessions) had NA = 16.4%. When we removed the 28(9.4%) that reported only “slight involvement” with the clinical teacher, there were no differences to the results in Table 2.

The Cronbach's α coefficient for the total CTAI scale was 0.95 with a mean standard error of measurement (SEM) across all the items =0.19. The CTAI, therefore, has very high internal consistency reliability and produces very small errors of measurement. The generalizability coefficient assessment provided an Ep² = 0.95 suggesting that the data provided to individual instructors was stable.

Factor analysis

The factor analysis showed that the data decomposed into three factors that accounted for 67.97% of the total variance (Table 3). The varimax rotation converged in seven iterations.

The factor loadings (bolded in square brackets for the large loadings) for each item on each factor are given in Table 3. From this pattern of loadings, it is evident that the factors together with the variance accounted for and their internal consistency reliability are Teaching skills (variance = 53.25%; Cronbach's α = 0.92), Patient interaction (variance = 8.56%; Cronbach's α = 0.91), and Professionalism (variance = 6.16%; Cronbach's α = 0.86). The three factors are intercorrelated (correlations = 0.48, 0.58, 0.46; p < 0.01; Table 3).

Discussion

The main findings of this study are: (1) a brief, 19–item, instrument was developed to assess clinical teaching based on the CanMEDS roles, (2) the CTAI had high internal consistency (Cronbach's α) reliability as did each of the factors, (3) the CTAI had high Ep² coefficients, (4) factor analysis of the CTAI resulted in three, theoretically meaningful, cohesive factors that accounted for nearly 70% of the total variance.

To create a brief instrument, we employed only 19 items. To carefully select the items, we surveyed a large number of published studies that reported on clinical teaching instruments. Our items were then carefully matched to the CanMEDS roles that are considered important physician characteristics. These procedures result in enhancing face and content validity of the CTAI.

Like other instruments (Irby & Rakestraw 1981; Irby 1983; Litzelman et al. 1998, 1999), the CTAI has high internal consistency reliability (Cronbach's α) and small SEMs. Additionally, we were able to show that with as few as eight raters, the scores on the CTAI are stable with an Ep² = 0.95. Both the internal consistency and generalizability across raters, then, are very high.

Beyond the face and content validity evidence, we examined the internal structure of the CTAI using factor analysis and were able to derive three cohesive, theoretically meaningful factors: Teaching skills, Patient Interaction, and Professionalism.

Teaching skills are central to any teaching assessment instrument (Beran & Violato 2009). Thus, the following items loaded on this factor: (1) Was well prepared for teaching sessions, (2) Organized time to allow for teaching and care giving, (3) Was stimulating, (5) Coached me on my clinical reasoning or technical skills, (6) Encouraged me to ask questions, (7) Incorporated research data or practice guidelines into teaching, (8) Emphasized a problem-solving approach rather than solutions, (9) Stimulated me to learn independently, (10) Clearly specified what I was expected to know and do during this rotation, and (11) Offered feedback. Meanwhile, other items did not load on Teaching skills but did on Patient Interaction: (14) Demonstrated compassionate patient-centered care, (15) Interacted effectively with patients and their families, (16) Answered questions clearly, (17) Taught effective patient/family communication skills, (18) Pointed out opportunities for health advocacy, (19) Responded to individual patient health needs as part of patient care. Finally, the remaining three items loaded on Professionalism: (4) Fostered an environment of respect in which I felt comfortable participating, (12) Was approachable for discussion, and (13) Treated team members in a professional manner. The clustering of these items as well as the pattern of loadings summarized in Table 3 provides both convergent and divergent validity evidence for the CTAI.

The results of this study support our goal of developing a brief instrument which is efficient and feasible and for which there is evidence of validity and reliability. The CTAI is simple to administer, takes a minimum amount of time for residents to complete, and is generally consistent with the culture of medical education at most universities. It also highlights the importance of CanMEDS roles for teachers and is expected to contribute to the continued dissemination of the CanMEDS roles into the medical teaching community at large.

There are several limitations to this study. In order to receive forthright data, it was important to insure the anonymity of resident responses. Therefore, it was not possible to link repeat evaluations of the same physicians by the same learners to evaluate intra-observer reliability. Moreover, the instrument may be of limited value for learners that have spent little time with the preceptor/teacher, although the psychometric properties of the instrument did not change in any way when we removed these low involvement learners from the analyses. The data from this study come from only one institution and it will be valuable to collect data at other medical centers to study the generalizability of our findings.

Summary and conclusions

The focus in this study was to investigate the reliability and validity of the CTAI. The psychometric results provided support for evidence of both reliability and validity. Nonetheless, we were not able to assess how the teachers responded to feedback from the instrument. It will be useful to create individual faculty reports and study the effect the feedback has on subsequent teacher behavior. Finally, future research may well focus on the relationship between scores on the CTAI and student learning. Meanwhile, the CTAI is a brief, simple to administer instrument with evidence for reliability and validity.

Declaration of interest: The authors report no declarations of interest.