The Academic Support Process (ASP) website: Helping preceptors develop resident learning plans and track progress

Abstract

Background: At times, preceptors struggle with aspects of resident education. Many are looking for more support and faculty development in this area.

Aims: To address preceptors’ needs for resources and provide a proactive framework for their teaching, the Academic Support Process (ASP) website was developed and evaluated. Preceptors’ (N = 35) experiences using the ASP website, as well as their perceptions of its usefulness in supporting resident education, were identified.

Methods: The research comprised two phases: a self-directed workshop involving the creation of a web-based learning plan for a standardised scenario of a resident in difficulty followed by 3 months use of the ASP website with residents in their practice. Information on their experiences was solicited via surveys and focus group interviews.

Results: Findings revealed the ASP website enabled preceptors to find words for their concerns around resident competency, gave them a proactive teaching framework, expanded their arsenal of teaching strategies, and supported a customised approach for all learners along the performance spectrum. However, there were a number of challenges encountered by the preceptors that affected site use and buy in.

Conclusions: Results are promising. Next steps involve developing a clear strategy for adoption.

 Introduction

In 2005, the Royal College of Physicians and Surgeons of Canada (RCPSC) released the CanMEDS framework as the basis for residency accreditation in Canada (Frank 2005). CanMEDS covers seven physician roles: Medical Expert, Communicator, Collaborator, Manager, Health Advocate, Scholar, and Professional. In 2009, the College of Family Physicians of Canada (CFPC) adapted CanMEDS to family medicine (Tannenbaum et al. 2009). CanMEDS-FM follows the CanMEDS 2005 structure with the replacement of Medical Expert with Family Medicine Expert to better reflect the ‘competencies required in the day-to-day practice of broad-based comprehensive and continuing care in family medicine’ (Tannenbaum et al. 2009, p. 3).

Postgraduate medical education has been reconfigured to accommodate this outcomes-based framework (McLeod & Steinert 2010). However, with the recent expansion of training positions (Ontario Ministry of Health and Long-Term Care 2005) and the corresponding increase in preceptors recruited, many preceptors responsible for supervising and evaluating resident learning have little theoretical background or training as educators and lack familiarity with the CanMEDS constructs. Al Shehri (2003) indicated, ‘The assumption that any good medical practitioner has the ability to teach no longer holds and professional training is required’ (p. 329). Indeed, postgraduate faculty have expressed both feelings of uncertainty and the desire for more training with regards to the development of resident competency within the CanMEDS roles (Verma et al. 2005; Frank & Danoff 2007).

Table

Practice points

The ASP website (http://uottawa.academicsupportplan.com) provides a framework to support preceptors in their resident teaching through the provision of educational resources and a learning plan development tool. Learning plans should be developed in collaboration with the learners. Preceptors desire faculty development to improve and support their teaching. Online tools require a careful balance between ease of use and richness of features. Web-based innovations require ongoing technical support.

In our roles training and evaluating residents, we have recognised that preceptors require particular support when supervising residents struggling to meet residency requirements. Early identification and articulation of residents’ poor performance is often extremely challenging for preceptors, as are implementing strategies for promoting change, identifying tools for the assessment of learning, and providing support for the resident (Dudek et al. 2005; Schipper & Ross 2010). Preceptors are often left to develop resident learning plans on their own without having the necessary skills, knowledge, or support. A learning plan is a written agreement between a learner and teacher that outlines learning expectations, strategies, and evaluation. The use of personalised learning plans is a proven effective educational approach (Cohen 1996; Brambleby & Coates 1997; Challis 2000).

To address the need for preceptor support and education around CanMEDS and resident supervision the Academic Support Process (ASP) website (http://uottawa.academicsupportplan.com) was developed to: (1) promote an academic, evidence-based teaching culture grounded in CanMEDS; (2) provide up-to-date evidence-based educational resources; and (3) provide preceptors with learning plan development and tracking tools that will increase their effectiveness and comfort as teachers when teaching residents with significant learning gaps. An online approach was chosen to facilitate access to the resources.

The purpose of this research was to identify preceptors’ experiences using the ASP website, as well as their perceptions of its usefulness in supporting resident education.

 The ASP website

The ASP website contains the following:

1. Educational resources.

There are four resource categories:

Issue identification: Lists family medicine resident competencies organised according to CanMEDS, with notations specific to International Medical Graduates (IMG) where appropriate. CanMEDS-FM had not been released when the original website was designed so we customised CanMEDS to family medicine based on a literature review and personal experience as family physicians. We chose to frame the website in CanMEDS for completeness and portability to other residency programs. Upon release of CanMEDS-FM, the content was reviewed for consistency and only minor changes were required.

Learning objectives: Provides basic information about learning objectives—what they are, why they are helpful, and how to write high-quality objectives. Lists useful verbs for writing appropriate learning objectives. Includes examples of common learning objectives for postgraduate learners.

Teaching strategies: Describes teaching strategies commonly used in medical education with best practice evidence. Includes a grid to help preceptors select appropriate teaching strategies to address the issues identified within each CanMEDS role.

Special considerations: Suggests resources for when residents need to take a leave, need support for mental health issues, or when a residency needs to be terminated.

1. Learning plan development tool.

Guides preceptors in the design of individualised 4-week (length of rotation) learning plans that are stored securely online and can be accessed by faculty support team members for that resident (e.g. primary preceptor, Unit Director, Director of Evaluation, and other faculty involved in the implementation of the plan). Users follow a step-by-step procedure to create the plan: (1) add resident demographic information, (2) indicate what led the resident to be identified as needing additional support, (3) identify the resident's strengths, (4) document one area of concern by identifying the CanMEDS role and describing the issue, (5) write a learning objective for the issue, and (6) select a suitable teaching strategy to achieve the learning objective and indicate the frequency of implementation and the support team member responsible for implementing and/or supervising the strategy. Steps 4–6 are repeated for each issue identified. Once completed, preceptors can automatically email a notification of the plan to the support team for feedback and approval. The email includes a link to the online learning plan and login instructions. While support team members cannot edit the plan directly, they can comment on the plan online using the comment box. In order to ensure confidentiality, support team members only have access to the learning plans of the residents in which they have been tagged as support team members.

1. Tracking tool.

The ASP website automatically generates a customised tracking form based on the learning plan developed. Each week the preceptor is tasked to track the use of the teaching strategies. In the middle and at the end of the plan, preceptors indicate the learner's progress towards each of the learning objectives. Through the online discussion tool, the preceptor and support team members can record their observations, document feedback shared with the learner and the learner's responses, record details of incidents, and offer specific suggestions regarding support thereby providing an ongoing record of the teaching–learning process and the resident's progress. At the end of 4 weeks, preceptors indicate whether the resident has met expectations or whether he or she needs further support. If further support is needed, the preceptor can reset the tracking and/or modify the plan for another 4-week period.

Currently, access to the learning plan and tracking tools is restricted to preceptors in the Department of Family Medicine at the University of Ottawa. However, due to demand, an open version of the website (http://www.academicsupportplan.com) was created that replaces the password protected database with learning plan and tracking templates that can be used by preceptors from all specialities from any institution.

 Methodology

 Procedures

Phase one

Preceptors attended a workshop where they completed a demographic survey and were introduced to the ASP website through a video (http://uottawa.academicsupportplan.com) and WebQuest. They then created a learning plan using the ASP tool for a resident in difficulty based on a standardised scenario. At the end of the workshop, preceptors were surveyed to identify their experiences using the website. The survey included qualitative and quantitative items and was based on the Demand-Driven Learning Model (DDLM) evaluation tool (MacDonald et al. 2002). The DDLM evaluation tool aligns with the DDLM (MacDonald et al. 2001), an eLearning framework.

Phase two

Following the workshop, preceptors were asked to use the ASP website for 3 months in their resident teaching, both as an educational resource and to develop learning plans for their residents and track their progress. After 3 months, preceptors were asked to complete another survey and invited to participate in a focus group interview to share their experiences using the website in the field.

 Data analysis

Descriptive statistics were calculated for the quantitative survey items. A paired sample t-test was used to compare scores on the surveys at the end of the workshop and after the in-field implementation. The constant comparative method was used to analyse the interview transcripts and the participants’ responses to the open-ended survey items. The transcripts were read and reread and a preliminary list of relevant emergent categories was developed. Once the categories were created satisfactorily, the data were assigned to the categories and the findings compiled into a report. Direct quotations were used to preserve the voice of the participants.

 Findings

 Participant characteristics

Thirty-five preceptors from the Department of Family Medicine at the University of Ottawa, representing a wide range of ages and teaching experience, completed phase one (51% male, 49% female). They had from 1 to 25 years experience teaching residents (mean = 9.97; SD = 7.08), with an average of 6 years as primary preceptors (SD = 5.30). Eighty percent (n = 28) of the preceptors indicated they were familiar with the CanMEDS-FM roles, yet only 40% specified they use CanMEDS-FM to frame their teaching and assessment. On a scale of 1 (low) to 5 (high), preceptors rated their knowledge (mean = 2.65; SD = 1.10) and use (mean = 2.03; SD = 1.01) of learning plans relatively low. However, they rated their computer abilities, attitudes, and comfort working online high (see Table 1). Fifteen preceptors completed the final survey and eight participated in the focus groups at the end of phase two.

Table 1  Preceptors’ computer abilities and attitudes

	Total
	Frequency	Percentage
Computer ability
Non-existent (e.g. I rarely use a computer and struggle to do simple tasks)	0	0
Basic (e.g. I can perform simple word processing tasks, conduct a web search, send an email)	3	9
Intermediate (e.g. I use a computer on a daily basis at work; use electronic health records; am comfortable with a variety of software applications; can add/remove programs)	29	83
Advanced (e.g. I have advanced computer knowledge and skills and am able to problem-solve computer-related problems)	3	9
Attitude towards computers
Very negative	0	0
Negative	1	3
Neutral	6	17
Positive	20	57
Very positive	8	23
Comfort working online
Not at all comfortable (e.g. when I go online I often get lost and find it hard to navigate the web)	0	0
Moderately comfortable (e.g. confident navigating the web, searching online)	14	40
Very comfortable (e.g. complete and submit online forms, use web-based software, shop online)	17	49
Extremely comfortable (e.g. store and share documents online, engage in real-time online collaboration [e.g. application sharing], engage in online learning)	4	11

 Use of the site

In the post-workshop survey, on a scale of 1 (low) to 5 (high), preceptors indicated a moderate intention to use the site (mean = 3.82; SD = 1.11). As the implementation period was only 3 months, preceptors were encouraged to use the tool with all their residents, not just those with issues of concern. Twenty-three learning plans were created by 15 preceptors. Preceptors reported they used the tool more to create learning plans than to track residents’ progress, and even less to communicate with support team members.

 The need for the ASP website

Support when supervising a resident in difficulty

Preceptors are looking for tools and resources to augment their teaching. They do not always have mentors to turn to for help and appreciate having easily accessible resources. Some indicated that prior to using the ASP site they did not know what to do when they had a resident in difficulty. In addition, based on prior experiences, the extant literature, and the interviews with preceptors, it is clear that preceptors struggle to identify the issues for residents in difficulty and put these into words. One preceptor described his struggles as follows:

I knew there was a problem but did not have ways to categorise the potential problems. [I was] observing behaviours that were clearly disturbing but not necessarily knowing what they meant. So, being able to translate behaviours into some sort of framework that I could then develop a management plan [for was needed]. (04B05J)

Not only was having a framework, such as the one provided by the ASP website, valued by the participants, but so was having a common language to describe these issues. As one preceptor noted, ‘I think it was difficult to find the words to put to the problem and know how that fit into categories. That has certainly been a useful part of this [site]’ (05P07E).

Prevent strong residents being ‘forgotten’

The tool also filled a need when teaching strong residents. A number of preceptors admitted that strong residents tend to get ‘forgotten’: ‘With a resident that is doing very well, you don’t have much to tell them. You say, ‘You’re doing very well. This was good. See you next month’’ (7574F). However, using the ASP tool prompted some preceptors with strong residents to structure their teaching for these learners:

I see that often in the Department, where the really good ones just don’t get challenged. Having to write a learning plan for a resident that isn’t having any problem has actually … helped me provide clear and concrete goals for this resident. (2529A)

Yet, what the preceptors want from the site for the two groups is different. For the strong residents, preceptors want to use the tool to identify specific, short-term learning objectives. For example, ‘Today … it looks like you had the wrong billing code, maybe you should read a bit about billing. I’m going to put this in as a learning strategy: work on billing’ (7574F). However, for a resident in difficulty they wanted something more complete. Preceptors indicated they were willing to invest more time to write more complete learning plans for the struggling residents than they were for strong residents.

Documentation

Preceptors felt that tracking the implementation of the plan and learner progress was an important feature for residents in difficulty but not for strong residents. As one preceptor said, ‘If you have a resident who's really struggling you are looking for clear documentation if there ever is a problem later on’ (09J01M). Conversely, for residents without formal concerns it was felt the time required to track their progress was not worth the effort.

Communication

The communication facilities of the ASP site were not used much. However, preceptors discussed the benefits of the ASP website as a communication tool. One preceptor felt that having a detailed and accessible online learning plan would help others better supervise, teach, and evaluate their resident on a sessional basis. Another cautioned about forward feeding and labelling – especially if it is a plan for an average resident, rather than one in difficulty.

Preceptors discussed the benefit of the communication tool when there are multiple people supervising individual residents and there is the need to share and track feedback, especially for In-Training Evaluation Report (ITER) purposes. By recording resident performance and behaviour online, pervasive issues can be distinguished from one off occurrences.

Another felt the tool could help them feel part of a team:

As a preceptor who only gets residents for rural blocks one month at a time, I would like to be able to add comments to a resident's permanent record. Partly because paper records get lost/ignored and partly because I would like to feel that I am part of a bigger team process.

 Preceptors’ experiences with the ASP website

Preceptors’ ratings of the ASP website were not significantly different after the in-field implementation to those after the workshop (p > 0.05). The data revealed a number of strengths of the website, as well as a number of challenges. Overall, preceptors indicated they enjoyed learning about learning plan development online using the ASP website, found the website engaging, and would recommend it to other preceptors (see Tables 2 and 3). Specific experiences related to content and design are addressed below.

Table 2  Preceptors’ responses after the workshop (N = 35)

	N	Min.^a	Max.	Mean	SD
Overall
I enjoyed learning about support plan development online using this website	34	1	5	4.03	1.01
The website was engaging	33	2	5	4.03	0.72
The website met or exceeded my expectations	33	2	5	3.82	0.80
I would recommend this website to other preceptors	33	2	5	4.06	0.69
I intend to use this website to create support plans for my residents	33	1	5	3.82	1.11
Content
The ASP website …
Includes content of appropriate depth and breadth	35	3	5	4.29	0.61
Is missing information that I would find useful when supporting residents	35	1	5	2.77	1.10
Provides clear and succinct information on learning strategies	35	3	5	4.14	0.49
Provides a comprehensive list of resident competencies within the Issue Identification section	35	1	5	4.31	0.78
Builds on my current experience and knowledge	35	2	5	4.09	0.77
Includes relevant information for preceptors	35	2	5	4.2	0.67
Makes important links between educational theory and practical teaching practices	35	3	5	4.11	0.71
Includes information that will allow me to improve my resident teaching skills	35	3	5	4.31	0.52
Includes information that will allow me to develop the skills necessary for creating a support plan	35	2	5	4.23	0.64
Design
The website is aesthetically pleasing	35	3	5	4	0.59
The website is clear and easy to read	33	3	5	4	0.65
The website is organized and well laid out	35	3	5	4	0.59
I could easily find the information I needed	35	2	5	3.46	0.73
It was easy to navigate throughout the website	35	3	5	3.77	0.59
It is easy to develop a support plan using the website	35	1	5	3.37	0.96
The website is sufficiently interactive	35	1	5	3.69	0.71
The icons, menu buttons, links, and controls did what I expected	35	1	5	3.74	0.87
I did not encounter any problems using the website	35	1	5	2.86	1.07
Outcomes
The website introduced me to new learning strategies	35	3	5	4.34	0.53
The website introduced me to useful teaching resources of which I was not previously aware	35	3	5	4.37	0.59
I acquired new skills and knowledge as a result of using this website	35	3	5	4.29	0.61
The website helped me identify and label specific issues for a resident in need (the resident in the scenario)	35	1	5	4.17	0.91
The website helped me identify appropriate learning strategies to help a resident in need (the resident in the scenario)	35	2	5	4.06	0.86
The grid helped me identify the most appropriate learning strategies for the issues identified (for the resident in the scenario)	35	2	5	3.97	0.91
The website helped me articulate clear and concrete goals for a resident in need (the resident in the scenario)	35	2	5	3.8	0.82
I feel more confident developing a support plan for a resident in need having used this website	34	3	5	4.15	0.55
The website meets my professional development needs in this area	34	3	5	3.88	0.63
I can see myself using what I have learned from this website with my residents	34	3	5	3.88	0.58

^aResponse options: 1 = Strongly Disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly Agree.

Table 3  Preceptors’ responses after the in-field implementation (N = 15)

	N	Min.^a	Max.	Mean	SD
Overall
I enjoyed learning about support plan development online using this website	15	1	5	3.80	1.17
The website was engaging	15	3	5	3.93	0.68
The website met or exceeded my expectations	15	2	5	3.53	1.02
I would recommend this website to other preceptors	15	2	5	4.07	1.06
I used this website to create support plans for my residents	15	3	5	3.73	0.68
I used this website to track resident progress towards the identified learning objectives	15	1	5	3.00	1.03
I used this website to communicate with support team members	15	1	4	2.60	0.95
Content
The ASP website …
Includes content of appropriate depth and breadth	15	3	5	4.13	0.62
Is missing information that I would find useful when supporting residents	15	1	4	2.60	0.95
Provides clear and succinct information on learning strategies	15	2	5	3.80	0.65
Provides a comprehensive list of resident competencies within the Issue Identification section	15	2	5	4.13	0.81
Builds on my current experience and knowledge	15	3	5	4.20	0.65
Includes relevant information for preceptors	15	3	5	4.33	0.60
Makes important links between educational theory and practical teaching practices	15	4	5	4.33	0.47
Includes information that will allow me to improve my resident teaching skills	15	3	5	4.27	0.57
Includes information that will allow me to develop the skills necessary for creating a support plan	15	4	5	4.40	0.49
Provides a useful tool for developing resident support plans	15	1	5	3.87	1.15
Provides a useful forum for tracking resident progress	15	1	5	3.6	1.2
Supports communication and collaboration between support team members	15	1	5	3.6	0.95
Design
The website is aesthetically pleasing	15	2	5	3.73	0.68
The website is clear and easy to read	15	3	4	3.67	0.47
The website is organized and well laid out	15	2	5	3.73	0.68
I could easily find the information I needed	15	2	5	3.40	0.80
It was easy to navigate throughout the website	15	1	4	3.20	0.91
It is easy to develop a support plan using the website	15	1	5	3.13	1.26
It is easy to track resident progress towards the identified learning objectives	14	1	4	3.21	1.08
It is easy to communicate with other support team members	13	1	5	3.31	1.07
The website is sufficiently interactive	14	2	5	3.64	0.72
The icons, menu buttons, links, and controls did what I expected	15	2	5	3.47	0.88
I did not encounter any problems using the website	15	1	4	2.47	0.81
Outcomes
The website introduced me to new learning strategies	15	4	5	4.53	0.50
The website introduced me to useful teaching resources of which I was not previously aware	15	4	5	4.53	0.50
I acquired new skills and knowledge as a result of using this website	15	4	5	4.47	0.50
The website helped me identify and label specific issues for the resident I am/was teaching	15	3	5	4.20	0.65
The website helped me identify appropriate learning strategies for the resident I am/was teaching	15	3	5	4.07	0.57
The grid helped me identify the most appropriate learning strategies for the issues identified for the resident I am/was teaching	15	3	5	4.07	0.68
The website helped me articulate clear and concrete goals for the resident I am/was teaching	15	4	5	4.53	0.50
The website facilitated communication between support team members	13	1	5	3.38	1.08
I feel more confident developing a support plan for a resident in need having used this website	15	2	5	3.87	0.72
The website meets my professional development needs in this area	15	1	5	3.67	1.07
I can see myself continuing to return to this website to access the resources	15	2	5	3.73	0.85
I can see myself continuing to use this tool to create resident support plans	15	1	5	3.53	0.96
I can see myself continuing to use this tool to track resident progress towards the identified learning objectives	15	1	5	3.60	0.95

^aResponse options: 1 = Strongly Disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly Agree.

Content

In terms of the ASP website content, preceptors’ responses on the surveys were positive (see Tables 2 and 3). Preceptors agreed the content was of appropriate depth and breadth, comprehensive, and relevant for preceptors, as well as that it built on their current knowledge and experience and made important links between educational theory and practice. Preceptors agreed the website included information that will allow them to develop the skills necessary for developing learning plans and improve their resident teaching skills. One preceptor indicated:

Great source of information and resources. [The depth] may at times be a barrier as it is not easy to just quickly navigate and do a learning plan. However, because of this, it is more likely that, if used, the learning plan will be better.

While still positive, preceptors’ responses regarding the utility of the site as a tool to develop learning plans, track resident progress, and support collaboration between team members were more neutral.

The data from the focus groups provided further insight into what the preceptors liked about the content of the ASP site. Exposure to different teaching strategies via the site was the most helpful aspect of the site for a number of preceptors; as one put it: ‘As an educator it helps me raise my game. I’m now learning these different techniques or different approaches; something other than ‘Let's see if we can get you more patients [with that diagnosis]’’ (04B05J). Another indicated, ‘For me, the main thing that was useful was just seeing what the different [teaching] strategies were. I thought, “Oh yeah, I never thought about doing that with that resident”’ (06D04E). The grid linking teaching strategies to issues identified according to CanMEDS role was also viewed very favourably.

The issue identification section was another strength of the site noted by participants:

With [a struggling] resident, when there are a lot of different issues it gets muddled up … . To actually pick it apart in a concrete manner and be able to say, ‘OK part of it is this, part of it is that. OK, what can I do?’ … helps me identify more clearly what the issues are and what the best strategies are to work with that. (2529A)

Design

The design and functionality of the site was rated less favourably than the content (see Tables 2 and 3). While the preceptors tended to agree that the website is aesthetically pleasing, clear and easy to read, and well laid out, their responses were more neutral around usability. Preceptors complained about technical glitches, as well as the time required to develop a learning plan. One preceptor explained:

I find the website very helpful in terms of content and information but I find making a learning plan on it very slow, too slow for the volume of work we have and because of that it's not user friendly. It is too detailed as far as the amount of things to enter.

One step the preceptors found particularly cumbersome was adding the information around the teaching strategies (i.e. indicating teaching strategy(ies), frequency of use, and individual responsible for implementing the strategy). In some cases, the intricacy of the tool impacted what preceptors put into the plan. One preceptor revealed: ‘I found myself trimming what I wanted to put as feedback because I didn’t want to spend much more time on it … . I don’t want to have ten things I want to tell them and then limit it to three because the structure makes it hard for me to [enter it]’ (04B05J).

While the framework for developing learning plans was appreciated by some preceptors, the step-by-step procedure was a challenge for others, as was the imposed structure of a 4-week plan. Preceptors questioned what learning objectives could be achieved in 4 weeks; ‘many of the learning objectives happen over such a longer period of time’ (2529A).

 Experiences related to teaching knowledge and approach

Some preceptor comments related less to the tool and more to their comfort to create learning plans and manipulate the learning environment. Two themes emerged:

Acknowledgement of roles and responsibilities as preceptors

At least five of the eight preceptors in the focus groups expressed concern about having to indicate the frequency they would implement the teaching strategies. One preceptor explained, ‘It reads very much like a contract and that's what makes me nervous … . I’m not sure that I can keep up my end of the bargain’ (04B05J). Another revealed, ‘There is probably the tendency to lowball it because you don’t want to look like you’re not doing your work’ (2529A). Preceptors were concerned that the unpredictability of the clinic (e.g. patient population during the block, call schedule and vacations) would prevent them from being able to do what they commit to in a learning plan.

Some preceptors argued that medical education is meant to be experiential and the idea of setting learning objectives does not fit with this. As mentioned by one preceptor, medicine is ‘teaching by opportunity. It's experience. It's different from learning multiplication tables. This is on the job, experiential learning … . It seems artificial to create objectives’ (05P07E). This approach to medical education will no doubt impact preceptors’ perceptions of the ASP tool where they are required to set learning objectives as part of the learning plan.

Knowledge gaps

For many of the preceptors, writing learning objectives and developing learning plans was a new experience and a challenge in itself due to a lack of medical education knowledge and skills. At least three preceptors reported finding it extremely difficult to write effective learning objectives. One noted, ‘It's pretty obvious [when] someone needs to work on their interview style or patient-centeredness but to write that as a [clear learning] objective I found that a bit of a challenge’ (06D04E). There was a clear demand for faculty development in this area.

Another challenge to writing the learning plans arose from a lack of knowledge of CanMEDS and the terminology surrounding it:

Part of the problem initially for me organising feedback for people, was the idea that I didn’t have a language for it … . There was all of this new language coming in; and I call it a language because I didn’t speak CanMEDS a couple of years ago … . It wasn’t how I thought of myself. I didn’t break down my activities in terms of ‘I’m being a manager right now. I’m being professional’. It was just part of the Gestalt of practice. (04B05J)

 Outcomes

There were a number of positive reported outcomes associated with the use of the ASP website (see Tables 2 and 3). There were no significant differences between participants’ survey responses after the workshop and after the in-field implementation (p > 0.05). Preceptors indicated the website introduced them to new teaching resources and strategies they could use with their residents and helped them acquire new knowledge and skills. Furthermore, they agreed that the website helped them identify and label specific issues for the residents they were teaching, articulate clear and concrete goals for them, as well as identify the most appropriate teaching strategies to use. While not with the same strength as noted above, preceptors also agreed that they felt more confident developing a learning plan for a resident in need having used this website.

Other benefits related to the fact that having a written plan agreed to by the resident made the resident more aware and accountable as an adult learner. As well, the site helped with ITERs as it provides preceptors with a broader vocabulary to describe issues, helps document specific examples of learners’ performances and behaviours, and gives more focus to discussion. As well, the ASP website has the potential to support a community of educators and prompted preceptors to share tools and talk to each other now that they had a common language with which to communicate.

 Looking to the future

In terms of using the ASP website in the future, preceptors were neutral to positive (see Tables 2 and 3). Preceptors tended to agree they would return to the website to access the resources but were more neutral regarding their intentions to continue to use the tool to create learning plans and track resident progress. The neutral attitude may be towards the development of learning plans in general, rather than the tool specifically, as demonstrated previously when learning plans were introduced at one unit. Nonetheless, at least three of the eight preceptors in the focus groups talked about using the learning plan development tool for all their residents at the start of the residency or in-unit block. Furthermore, all three indicated they would develop the plan in collaboration with the resident. One preceptor planned to send her residents to the site so they could have access to the information and start to put some thought into the learning plan before meeting with her. She indicated that would be ‘helpful, as then we’d also be using the same language and you’re more likely to have buy-in if they [identify what they want to work on themselves]’ (09J01M). Others agreed. Indeed, preceptors talked about the importance of introducing the site to the residents at the start of their residency and ‘talking about how this is collaborative learning’ (09J01M).

There was a strong sense among the preceptors that this type of tool is needed in the Department of Family Medicine. Reasons included a need for consistency and high quality ongoing evaluations, to provide residents with transparent goals and expectations, as a faculty development tool to guide teaching, and because it provides a standardised means of evaluating residents and developing learning plans. However, preceptors voiced the importance of avoiding duplication of other systems and/or documentation required in their roles as preceptors. Preceptors also highlighted ways the tool could be expanded to facilitate teaching and evaluation. Preceptors want to be able to record their direct observations using the ASP tool and collate examples of residents’ performances to guide their evaluations. The need to integrate the ASP website with other departmental systems was stressed.

As this tool continues to be introduced within the Department of Family Medicine there is need for faculty development. Preceptors are looking for help writing clear and measurable learning objectives, selecting appropriate teaching strategies, and documenting their proactive approach adequately. As well, they are looking for feedback on the plans they create to improve their skills, knowledge, and effectiveness.

 Discussion

The ASP website was developed in a modest attempt to address preceptor needs for support and training around resident education. The need for the ASP website was reinforced by the data and our efforts were welcomed and encouraged by the preceptors involved. Preceptors felt the ASP website helped them put words to the concerns they had around resident competency and gave them a framework to deal with those issues. It also helped them be more diligent teachers when working with strong residents and expanded their arsenal of teaching strategies. While not used to a great extent, preceptors appreciated the ability of the tool to document learners’ progress and facilitate communication between the support team members. The use of these aspects of the tool, once the ASP website is implemented throughout the Department, is worthy of further examination.

Overall, the content on the site was perceived positively by the preceptors. However, there were a number of issues around the usability of the site that impacted some preceptors’ use of the learning plan development and tracking tools. Preceptors found the tools time consuming to use, partly as a result of some technical glitches that had not been remedied at the time of the evaluation and partly because of the length of time required to develop the content of a learning plan, their lack of experience creating plans, and a limited background in medical education. The importance of proper orientation to the website through faculty development and an appreciation of the time dependent learning curve by the departmental leadership will be essential for further acceptance and buy in.

The technical glitches experienced point to the need for responsive technical support when undertaking such a complicated web-based database-driven innovation. While the technical team were enthusiastic and engaged developing the site, it became clear as the project was implemented and bugs surfaced that a continuous investment in skill, time, and commitment is required. The importance of determining appropriate and available resources and defining a clear service agreement from the start is critical in these types of projects. Decisions around in-house versus external development are important considerations. While in-house might be a cheaper option there are likely to be a number of competing priorities for resources.

The weightiness of the tool had advantages and disadvantages. Requiring preceptors to use a strict step-by-step process meant that they had to dissect each issue, create a specific learning objective for that issue, and then determine how they were going to address it with the resident—all sound pedagogy. However, this process is time consuming and time is a precious commodity for busy clinician preceptors. While the ASP website was initially developed to support preceptors supervising residents with learning gaps, the majority of the preceptors used the tool with well performing residents in this evaluation. While it is likely easier to learn the academic support process when teaching residents with minor learning gaps, it may explain why some preceptors felt the tool was too time consuming. The ASP approach also required preceptors to be linear in their thinking. The templates on the open site, created using Microsoft Word, allow for more freeform entry of the issues and it is our understanding that some preceptors have started to use these rather than the more rigid online tool. An evaluation of the open site with preceptors outside family medicine and across diverse institutions is a worthy next step.

Preceptors’ concerns around committing to specific teaching strategies in a learning plan, lack of comfort writing learning objectives, and unfamiliarity with CanMEDS-FM strongly highlights the need for a greater emphasis on faculty development. While the ASP website can play an important role in changing attitudes and culture and providing the required resources, it is only one piece of the puzzle. It is critical to ensure that preceptors are aware of the specified learning objectives for the clinical rotations and have a clear understanding of their roles and responsibilities as preceptors.

The results of this evaluation are promising. The next step is to consider the level of the tool's integration into Department processes. There will need to be a clear strategy for how the tool will be introduced and integrated into the roles and responsibilities of the preceptors in order to maximise adoption. The use of John Kotter's (Kotter 1996; Kotter & Cohen 2002) eight-step change model is worthy of consideration to guide and inform this integration. Including the use of the ASP website in faculty performance appraisals, mandating the development of a learning plan to access additional funds to support residents in difficulty, providing residents with access to the site to allow them to initiate the development of a learning plan, and requiring residents to have learning plans for all rotations are just some suggestions to facilitate the use of the tool and help establish an academic evidence-based culture around teaching that encourages a learner-centred CanMEDS approach.

By grounding ASP in the faculty development arm of the Department of Family Medicine and through the inclusion of the website in an open access provincial faculty development repository (R-Scope), we hope to meet our goals for nurturing an increasingly vibrant medical education community of practice.

Acknowledgements

The authors would like to thank Dr Meridith Marks for her comments on an earlier draft of this manuscript.

Declaration of interest: This project was funded by an Educational Initiatives in Residency Education Grant, Postgraduate Medical Education; a Department of Family Medicine Program for Innovation in Medical Education (PIME) Grant; the Department of Family Medicine; and the Academy for Innovation in Medical Education, University of Ottawa.