The evaluation of learner outcomes in interprofessional continuing education: A literature review and an analysis of survey instruments

Abstract

Background: Interprofessional education (IPE) is thought to be important in fostering interprofessional practice (IPP) and in optimizing patient care, but formal evaluation is lacking.

Aim: To identify, through review of IPE evaluation instruments in the context of Barr/Kirkpatrick's hierarchy of IPE learner outcomes, the comprehensiveness of current evaluation strategies and gaps needing to be addressed.

Methods: MEDLINE and CINAHL were searched for work relating to IPE/IPP evaluation published between 1999 and September 2010 that contained evaluation tools. Tool items were stratified by learner outcome. Trends and gaps in tool use and scope were evaluated.

Results: One hundred and sixty three articles were reviewed and 33 relevant tools collected. Twenty-six (78.8%) were used in only one paper each. Five hundred and thirty eight relevant items were identified, with 68.0% assessing changes in perceptions of IPE/IPP. Fewer items were found to assess learner reactions (20.6%), changes in behaviour (9.7%), changes in knowledge (1.3%) and organizational practice (0.004%). No items addressed benefits to patients; most were subjective and could not be used to assess such higher level outcomes.

Conclusions: No gold-standard tool has been agreed upon in the literature, and none fully addresses all IPE learner outcomes. Objective measures of higher level outcomes are necessary to ensure comprehensive evaluation of IPE/IPP.

Introduction

In the past decade, interprofessional practice (IPP) has increasingly been touted to be the gold-standard approach to optimal patient care (Barr 2005; D’Amour & Oandasan 2005; Zwarenstein et al. 2005). With budgetary and staffing shortages in the recent years, more efficient and effective use of health human resources have been encouraged through governmental and institutional initiatives (Canadian Interprofessional Health Collaborative 2008; Ontario Health Quality Council 2009). These have in turn supported a great deal of research in the areas of IPP and interprofessional education (IPE) – widely assumed to improve IPP (Curran et al. 2005; Stone 2006).

IPE is defined by the Centre for the Advancement of Interprofessional Education as being ‘occasions where two or more professions learn with, from and about each other to improve collaboration and the quality of care’ (Centre for the Advancement of Interprofessional Education 2002). Implicit in this statement is the concept of IPE as the means to further the ends of IPP, defined as the ‘provision of comprehensive health services to patients by multiple health caregivers who work collaboratively to deliver quality care within and across settings’, as well as improved patient outcomes. While the theoretical and preliminary support for this has been established (Curran et al. 2005; Stone 2006), work is required to evaluate the outcomes of IPE. It is only through such evaluation that an evidence-based approach to the implementation of IPE, and its impact on IPP, can be established.

Kirkpatrick (1996) provides a rigorous framework to evaluate learner outcomes of educational initiatives, and Barr (2005) has adapted this to suit the context of IPE (Table 1; Freeth et al. 2005). Conceptualized as a hierarchy of outcomes, the first level of the hierarchy is reactionary, as learners evaluate the learning experience and how they felt about it, and could be aligned with the learner's satisfaction with the IPE initiative. The second level encompasses changes in attitudes and perceptions towards IPP and other professional groups (2a), as well as what the learners gained from the experience in terms of related knowledge and skills (2b). The third and fourth levels of the hierarchy extend beyond the individual to how, if at all, the experience changed the learners’ approach to professional practice, and finally how that might impact organizational structure and patient outcomes.

Table 1.  Kirkpatrick/Barr's hierarchy.

Level 1a: Reaction

Learners’ views on the learning experience and its interprofessional nature

Level 2a: Modification of attitudes/perception

Changes in reciprocal attitudes between participant groups. Changes in perception or attitude towards the value and/or use of team approaches to caring for a specific patient/client group.

Level 2b: Acquisition of knowledge and/or skills

Including knowledge and skills linked to interprofessional collaboration.

Level 3: Behavioural change

Identifies individuals’ transfer of interprofessional learning to their practice setting and their changed professional practice

Level 4a: Change in organizational practice

Wider changes in the organization and delivery of care

Level 4b: Benefits to patients/clients

Improvements in health or well-being of patients/clients

While Kirkpatrick/Barr's taxonomy of learner outcomes is used frequently in the literature (Carpenter et al. 2006; Curran et al. 2007; Gillan et al. 2010), a number of other models and approaches have been used (Kramer & Schmalenberg 2003; Chakraborti et al. 2008). The complexity and lack of consensus of the relevant factors that signal success of IPE outcomes contribute to the relative inattention to some of the higher level outcomes in the literature (Hammick et al. 2007; Reeves et al. 2008). In addition to the diffuseness of outcomes used in the IPE literature, there is also a lack of consistency in measures used to evaluate these outcomes (Reeves et al. 2008). Without a widely accepted instrument, many programmes have resorted to the development of ‘one-off’ solutions. The authors of one study stressed the need for reliable, valid measures, concluding that these are ‘essential to establishing the role of IPE in health professions education’ (Zwarenstein et al. 2005). Thannhauser et al. (2010) reviewed 23 quantitative measures found in the interprofessional literature, and found that most ‘lack[ed] sufficient theoretical and psychometric development’ (p. 336).

This literature review was undertaken with the intention to establish a clear picture of the number, scope and characteristics of IPE evaluation instruments employed in the literature. Existent questionnaire instruments will be broken down into individual items and classified according to Kirkpatrick/Barr's hierarchy of learner outcomes, thus highlighting any outcomes that tend to be underrepresented in the literature, or identifying where good tools exist. This review will serve as a platform for the planned development of a comprehensive IPE evaluation tool.

Table

Practice points

IPE evaluation questionnaires can provide straightforward means to evaluate IPE initiatives and their outcomes, but the measurement properties of most are weak While a number of tools exist, none has been adopted as the gold standard and none is comprehensive in its assessment of all levels of Kirkpatrick/Barr's IPE learner outcomes There is a pressing need for comprehensive evaluation, in part through more objective assessment of higher level outcomes A single comprehensive tool is not feasible, and an IPE evaluation questionnaire toolkit with both standardized and tailored items may be more appropriate to address different contexts

Methods

Phase 1: Literature search and initial instrument classification

The MEDLINE and CINAHL databases were used to search keywords relating to the evaluation of IPE and practice. The search was restricted to work published between 1999 and June 2009, with the assumption that any relevant instruments published earlier would be captured through citation in other, more recent studies. A list of keywords was created by investigators in collaboration with an on-site medical librarian, and was informed by the search criteria defined by the 2008 IPE Cochrane Review (Reeves et al. 2008). Criteria were further refined beyond generally accepted IPE search criteria through inclusion of terms relating to education evaluation, such as evaluation, questionnaire, instrument and outcome. An initial search result suggested the value of focusing on those publications indexed as clinical trials, multi-centre studies, or research or reviews, in order to eliminate publication types that did not tend to be applicable. An example of the search strategy is included as Figure 1.

Figure 1. Search strategy. Database: Ovid MEDLINE(R) <1950 to June Week 2 2009> Search Strategy.

Once the MEDLINE strategy was finalized, this was then adapted to search CINAHL using appropriate CINAHL headings and keywords in the appropriate combination. The search of both databases was updated through September 2010 prior to completion of data analysis.

Citations were collected using EndNote, and were categorized preliminarily and independently by two investigators as (1) not relating to IPE, (2) relating to IPE, but not referring to the evaluation of IPE outcomes, (3) referring to IPE outcomes and/or instrument(s) and (4) including an instrument or instruments relating to IPE outcomes. A third investigator, using the same criteria, addressed any lack of consensus on initial categorization.

Phase 2: Instrument retrieval

Full-text articles for all abstracts in categories 2, 3 and 4 were reviewed by the PI for the inclusion or mention of instruments relating to IPE or IPP. Identified instruments were sought through PubMed, MEDLINE, Google Scholar, Scholars Portal, and JSTOR Arts and Sciences I Collection. Some complete instruments were included, either within the text or as appendices, in the reviewed papers. Other instruments were found by searching instrument titles and keywords in the above online databases and by searching any original references given for an instrument in the reviewed paper.

Phase 3: Item categorization

Instruments were broken down into individual items by two investigators, and the items were stratified independently by each investigator according to Kirkpatrick/Barr's hierarchy, using the learner outcome definitions outlined in Table 1. Data for all items were collected, including: instances of use in reviewed papers, reliability and validity, instrument(s) of origin and response format.

Results

A summary of the breakdown of findings for all phases is shown in Figure 2.

Figure 2. Literature search.

Phase 1: Literature review and initial instrument classification

A total of 1662 abstracts were identified using the search criteria, of which 24 were duplicates, leaving 1638 abstracts to be reviewed. Two independent reviewers reviewed 1301 (79.4%) of the 1638 abstracts and agreed on the initial categorization of 1098 (84.4%) of majority sample. The third reviewer acted as a tie breaker for the remaining 203 abstracts in the sample, and a consensus was reached through majority vote or discussion. The remaining 338 abstracts, representing the updated search for June 2009 through September 2010, were categorized independently by the PI. The ultimate categorization of all 1638 abstracts left 1436 (87.7%) in Category 1, 59 (3.6%) in Category 2, 84 (5.1%) in Category 3 and 59 (3.6%) in Category 4.

Phase 2: Instrument retrieval

After considering the number of articles found in Phase 1, investigators included all articles in Categories 2–4 (n = 202) in Phase 2, based on the anticipated likelihood of identifying instruments in each of the categories. Full-text articles were found for 184 (91.1%) of all 202 abstracts, with 163 of 184 (88.6%) found electronically, and the remaining 21 (11.4%) found in hard copy. This was felt sufficient, and no further means were undertaken to obtain the full text of the outstanding articles. The full text was reviewed for all articles and three further articles were subsequently eliminated from the review for not being relevant to IPE.

Analysis of the full-text articles yielded 44 IPE instruments, either referenced in-text and obtained elsewhere, or included in part or in their entirety as appendices. Eleven (25.0%) of the instruments could not be located or retrieved using the search strategy. Subsequently, 33 complete, relevant instruments were located and collected.

Of the 33 instruments that were found, 26 (78.8%) were mentioned and used in only one paper, 3 (9.1%) in two papers and 4 (12.1%) in 3 or more papers. The most frequently used instruments were the Attitudes Towards Healthcare Teams Scale (Hyer et al. 2003), the Interdisciplinary Education Perception Scale (Luecht et al. 1990) and the UWE Interprofessional Questionnaire (Pollard et al. 2004), which were each employed in four papers. Five of the reviewed papers made use of more than one identifiable instrument, with four papers (Pollard et al. 2004; Carpenter et al. 2006; Kwan et al. 2006; Loutzenhiser & Hadjistavropoulos 2008), each using two instruments, and one (MacDonald et al. 2008) using three instruments.

With respect to the timing of the use of the instruments within the context of the intervention or training programme being studied in each paper, 7 (21.2%) of the instruments were used exclusively post-intervention and 16 (48.5%) were used both pre- and post-intervention. For the remaining nine (27.3%) of the instruments, the time of administration relative to the intervention/training programme was not specified, most often when the methodology of the reviewed paper did not make use of an educational intervention. In these instances, study methods tended to involve control groups who were assumed to represent the pre-intervention state, or evaluations of the current state of practice of a clinical programme.

Both reliability and validity measures were evaluated in detail for seven (22.6%) of the instruments (Table 2). Constructs were discussed in detail for seven (22.6%) and nine (29.0%) of the instruments in terms of reliability and validity, respectively. Reliability was not addressed at all in 13 (41.9%) of the articles in which the instrument was found, and validity was not addressed in 15 (48.4%), with neither one being mentioned in any respect for 10 (32.3%) of the instruments. For the remaining instruments, either reliability, validity or both, was evaluated to some degree, but not comprehensively.

Table 2.  Item categorization by instrument.

Instrument	Origin	Total Items	Items excluded	Level 1	Level 2a	Level 2b	Level 3	Level 4a	Level 4b	Reliability	Validity
Attitudes Toward Healthcare Teams Scale	Heinemann et al. (1999)	21	0	0	21	0	0	0	0	F	F
Behaviour Change Collaborative Activities Index	Misra et al. (2009)	5	2	0	0	0	3	0	0	S	N
Collaborative Behaviours Scale	Stichler (1989) Source: Boone et al. (2008)	20	0	0	20	0	0	0	0	S	S
Collaboration and satisfaction about care decisions	Baggs (1994)	9	1	0	5	0	3	0	0	S	F
Demand-Driven Learning Model Evaluation Survey	MacDonald et al. (2002)	24	4	20	0	0	0	0	0	F	F
Healthcare Stereotypes Scale	Carpenter (1995)	24	0	0	24	0	0	0	0	N	N
Interdisciplinary Education Perception Scale	Luecht et al. (1990) Source: McFadyen et al. (2007)	18	0	0	18	0	0	0	0	S	S
Jefferson Scale of Attitudes Towards Nurse-Physician Collaboration	Hojat and Herman (1985) Source: Sterchi (2007)	15	0	0	15	0	0	0	0	S	S
Jones and Way's Collaborative Practice Survey	Way et al. (2001) Source: MacDonald et al. (2008)	9	0	0	7	0	2	0	0	N	N
Nurse–physician involvement in pain management activities	Kaasalainen et al. (2007)	36	24	0	0	0	12	0	0	N	S
Nurse–Physician Collaboration Scale	Ushiro (2009)	27	0	0	11	0	16	0	0	F	F
RIPLS	Parsell and Bligh (1999)	19	0	0	19	0	0	0	0	F	F
Role Conflict Questionnaire	Rizzo et al. (1970)	30	24	0	5	0	1	0	0	F	F
Team Climate Inventory	Anderson and West (1998)	61	7	0	48	0	5	1	0	F	F
Team Collaboration Index	Aram et al. (1971)	16	0	0	16	0	0	0	0	S	N
Trainee Test of Team Dynamics	Hyer et al. (2003)	6	0	0	2	4	0	0	0	N	F
UWE Interprofessional Questionnaire	Pollard et al. (2004)	35	0	0	35	0	0	0	0	S	S
Untitled	Carpenter et al. (2004)	40	21	18	0	0	1	0	0	N	N
Untitled	Chan et al. (2009)	14	5	9	0	0	0	0	0	N	S
Untitled	Clark (1994) Source: Dobson et al. (2009)	16	0	16	0	0	0	0	0	N	N
Untitled	Cooper et al. (2009)	3	0	1	2	0	0	0	0	N	N
Untitled	Crutcher et al. (2004)	39	19	4	15	0	1	0	0	S	N
Untitled	Dagnone et al. (2008)	10	2	7	1	0	0	0	0	N	N
Untitled	Garrard et al. (2006)	14	3	2	2	0	6	1	0	S	S
Untitled	Garvey et al. (2000)	10	4	6	0	0	0	0	0	N	N
Untitled	Hallikainen et al. (2007)	24	0	21	3	0	0	0	0	S	N
Untitled	Hewstone et al. (1994)	34	3	6	25	0	0	0	0	N	N
Untitled	Le (2008)	12	0	0	12	0	0	0	0	F	F
Untitled	MacDonald et al. (2008)	16	0	0	16	0	0	0	0	N	N
Untitled	Mann et al. (2009)	15	2	13	0	0	0	0	0	S	N
Untitled	Morison and Jenkins (2007)	25	2	0	18	3	2	0	0	N	N
	Total	663	125	111	366	7	52	2	0

Notes: F = assessment was adequately addressed, S = assessment was somewhat addressed, N = assessment was not addressed.

Phase 3: Item categorization

A total of 663 unique items were identified from the collected instruments. One hundred and twenty five (18.9%) of these were deemed irrelevant to the evaluation of IPE or IPP, and were excluded from further consideration. These items concerned specific medical content or non-IPE related aspects of the educational intervention.

A total of 538 items were thus included in the analysis. The average number of items per instrument was 20.7 (range 3–61). Four hundred and sixty five (86.4%) of these 538 items were evaluated on a Likert scale, with others assessed through multiple choice (3.0%), yes or no (4.1%) or open-ended (6.5%) questions. The categorization of items proved less straightforward than initially anticipated, and investigators met on a number of occasions to discuss and fine tune the classification strategy and criteria, to stay as true as possible to Kirkpatrick/Barr's hierarchy. A decision was made to assign each item to the category thought most reasonable, and to keep a record of general rationale and common issues. The small sample of items reviewed independently by four investigators suggested a high degree of consensus on item classification, and ultimate categorization was agreed upon by two investigators and reviewed with the others. The final item classification is summarized in Tables 2 and 3.

Table 3.  Item categorization.

Levels in Barr's hierarchy	Number of items (%)	Number of instruments with items measuring this level (%)
Level 1: Reaction	66 (14)	7 (27)
Level 2a: Modification of attitudes/perceptions	338 (73)	22 (85)
Level 2b: Acquisition of knowledge and/or skills	7 (2)	2 (8)
Level 3: Behavioural change	51 (11)	10 (38)
Level 4a: Change in organizational practice	2 (0.5)	2 (8)
Level 4b: Benefits to patients/clients	0 (0)	0 (0)

The majority of items (366 of 538, 68.0%) were assigned to Level 2a, assessing changes in attitudes and perceptions relating to IPE and IPP; 76% of the 33 instruments contained items that assessed this level. Examples of items, all employing Likert scales, included ‘Patients ultimately benefit if healthcare professionals work together to solve patient problems’ from the Readiness for Interprofessional Learning Scale (RIPLS; Parsell & Bligh 1999), ‘We problem solve together’ from the Collaborative Behaviours Scale (Stichler 1989) and ‘My skills in communicating with other health and social care professionals would be improved through learning with students from other health and social care professions’ from the UWE Interprofessional Questionnaire (Pollard et al. 2004). Some items assigned to Level 2a, such as ones concerning the perceived function of nurses as support for doctors, were found in numerous instruments with slight variations in wording (Morison & Jenkins 2007), Jefferson Scale of Attitudes Towards Physician-Nurse Collaboration (Hojat & Herman 1985) and the RIPLS (Parsell & Bligh 1999).

As summarized in Table 3, 111 of 538 items (20.6%) were found to measure learner reactions (Level 1), and these represented items from 12 different instruments. As an example, one 24-item instrument, with 20 items thought relevant to IPE (Demand-Driven Learning Model (DDLM) Evaluation Survey (MacDonald et al. 2002) was composed entirely of Level 1 items, and included such items as ‘The content included information that I will be able to use to deal with new situations at work’, and ‘The learning resource respected my experience’, both also employing Likert scales.

Only seven of the 538 items (1.3%) were assigned directly to Level 2b, which represented the acquisition of skills and knowledge relating to IPE and IPP. All were open-ended, qualitative items assessing elements of shared learning (Morison & Jenkins 2007), or effective versus ineffective team behaviours (Trainee Test of Team Dynamics; Hyer et al. 2003).

Level 3 was addressed in 11 of the 33 instruments (33.3%) with a total of 52 items. Many of these items referred to observable and measurable actions, such as discussing, planning and liaising, though in no cases were there specific references to how a change in these actions would be quantified objectively. There were only two items, in two different instruments, that were found to be relevant to the assessment of organizational change (Level 4a). One of these, from Garrard et al. (2006), related to staffing changes after an educational intervention, and the second, from the Team Climate Inventory (Anderson & West 1998), was ‘Does the team have clear criteria which members try to meet in order to achieve excellence as a team?’. One paper described a tool that could not be included in this review because items were not sufficiently defined, but the design allowed for assessment of Level 4a outcomes (Mann et al. 2009). It involved learners developing their own items concerning expected practice changes. At 3 months post-intervention, learners would then be asked to rank the degree to which each of these individually-proposed changes had occurred. No tools or items addressed patient outcomes (Level 4b).

Overall, 17 instruments (51.5%) contained items relevant to only one level of Barr's hierarchy. Eleven instruments (33.3%) had items addressing two levels, and 3 instruments (9.1%) had items addressing three levels. Only the instrument by Garrard et al. (2006) with only 11 relevant items (of 14 total) was found to have items that addressed Levels 1, 2a, 3 and 4a.

Discussion

The importance of evaluating the outcomes of IPE initiatives has been recognized, as suggested by the degree to which the topic continues to be discussed in the literature. This literature review demonstrates that efforts are being made to evaluate IPE, but the variety of instruments used, the lack of established reliability and validity and the reliance on subjective measures to assess higher level outcomes – if they are being assessed at all – suggests that further work is required in this area.

Given the variety of instruments that have been developed, and that fewer than one-third were used on more than one occasion in the reviewed literature, it is apparent that no single instrument has yet been adopted as the gold standard for IPE evaluation. Many were developed to meet the needs of a specific programme. Some studies made use of multiple existent instruments, often only selecting sections that were thought relevant to the purpose at hand, thus negating any prior validity or reliability measures of the individual instruments, and further fragmenting the body of knowledge generated from these evaluations. This further suggests that no single instrument offered a sufficient solution to many educators and research teams.

Assessment of outcomes

Consideration of the many instruments employed in the literature suggests that there are a number of commonalities – many of them are common weaknesses – between them, and also that a number of gaps in the assessment of learner outcomes exist, according to Barr's hierarchy.

The degree of satisfaction with an IPE initiative (Level 1) was well-considered in some instruments, but not in the majority. Assessments of learner satisfaction tend to be done ad hoc and in isolation from the investigation of higher level outcomes. Barr et al. (1999) note that many early evaluations of IPE outcomes were actually ‘little more than feedback on students’ satisfaction’ (p. 537), and that there was poor consideration of other outcomes. Such feedback is valuable and relatively easy to collect using a questionnaire format, but work is still required to standardize a tool and to establish reliability and validity for it.

The evaluation of changes in perception (Level 2a) proved less straightforward than determining learner satisfaction with an educational initiative. While the majority of items found in this review were assigned to Level 2a, it was this categorization that was most contentious for the reviewers. Strict observance of Kirkpatrick/Barr's definition of Level 2a would dictate that it is the change in attitudes and perceptions relating to interprofessionalism that constitutes an IPE learner outcome. This would require items to be considered using a pre-/post-methodology, unless one invited the learner's self-reported perception of any change in attitude or perception. This was not always the case in the reviewed papers. For this reason, many of the items assigned to Level 2a in this review could not actually have been considered to evaluate changes in perception in the way in which they were employed in the reviewed paper.

In many instances, it was also apparent to the reviewers that items were likely intended to evaluate higher level outcomes, but that reliance on self-reporting of these outcomes was prohibitive of these items actually being considered for inclusion in Levels 2b, 3 or 4. Hammick et al. (2007) acknowledged the tendency towards self-reported behaviour change in published literature, noting that it ‘must be regarded as [a] weak approach to measuring behavioural change’ (Hammick et al. 2007). The value of assessing the impact on practice and on the care of the patient were often acknowledged in the reviewed instruments, but items such as ‘Developing a patient care plan with other team members avoids errors in delivering care’ (Attitudes Towards Healthcare Teams (Heinemann et al. 1999) and ‘Good teamwork has a positive impact on patient care’ (Morison & Jenkins 2007)) are framed in such a way as to assess perceptions as opposed to objectively measurable changes in patient outcome, and could thus only be useful in evaluate lower level outcomes.

There were only a handful of items, from two different instruments, considered to be objective measures of knowledge acquisition relating to IPE and IPP, despite the potential for this to be a relatively straightforward outcome to assess in certain areas. Reviewers identified poor item design as one of the primary reasons for this. In some instances, items included in Level 2a could be reworded or assessed using a multiple choice, true or false or open-ended format as opposed to a Likert scale to make them more conducive to objective assessment, and thus more appropriate for Level 2b.

In terms of changes in behaviour, more items were identified than were initially anticipated by reviewers, but of the 52 items found in this review that were ultimately assigned to Level 3, very few were framed in such a way to strictly encourage objective measurement of the change in behaviour. This was thought to relate in part to anticipated difficulty in collection and analysis of such data through use of a self-reporting questionnaire. Reviewers made their decision to include items here on the basis of whether it was felt that the given item could be measured objectively. For example ‘The nurses and physicians share information to verify the effects of treatment’, from the Nurse-Physician Collaboration Scale (Ushiro 2009), was included in Level 3, as opposed to ‘Individuals in other professions respect the work done in my profession’ from the Interdisciplinary Education Perception Scale (Luecht et al. 1990), which was included in Level 2a. It was felt that the sharing of information could be observed and measured through the review of departmental documentation, while the measurement of respect might prove more difficult. Nonetheless, no such quantification was explicitly stated to be required for Level 3 items found in this review.

The two items identified as assessing organizational change as an outcome of IPE concerned relatively front-line organizational change – staffing and team criteria. While both were potentially quantifiable, there were no guidelines as to what might constitute a ‘staffing change’ or ‘clear team criteria’, and the items therefore relied on the learner's interpretation of the significance of a change. This was further reflected in the fact that the item from the Team Climate Inventory (Anderson & West 1998) concerning team criteria was assessed on a Likert scale, thus suggesting that there might be varying degrees of ‘clear criteria’, and no clear cut threshold of change. Overall, neither organizational change nor changes in patient outcome (Level 4b) were addressed to any discernable degree in any of the reviewed instruments, beyond learner perceptions of these outcomes.

Interpretation and implications

No single survey or questionnaire instrument was found in this review that could act as a stand-alone instrument to assess learner outcomes across Kirkpatrick/Barr's hierarchy. In fact, no instrument seems to be in use that includes items which would assess the higher levels to any significant degree at all.

The authors often found it difficult to determine the evaluative intent of a given item. A review of best practice in the use of questionnaires amongst healthcare staff (McColl et al. 2001), provided the conclusion that ‘question wording and framing, including the choice and order of response categories, can have an important impact on the nature and quality of responses’, and Thannhauser et al. (2010) further suggested that, despite the need for appropriate IPE and IPE evaluation tools, the psychometry of existing tools lacked rigour. The practice of selecting relevant sections of multiple questionnaires to suit the purposes of a given evaluation jeopardizes the reliability, validity and generalizability of resultant instruments. Questionnaires are often chosen as a methodology, however, because they are seen to be convenient, easily tailored to the need at hand and do not require a complex statistical analysis (Walonick 1993).

As well as being difficult to assess objectively through questionnaires, investigators recognized over the course of this review that higher level outcomes tend to become more specific to the individual practice environment as opposed to the educational initiative. For that reason, it becomes increasingly difficult at higher levels to develop items and instruments that would be generalizable beyond a given programme, specialty or practice environment. Some studies have made use of statistics relating to time to discharge, patient scores on psychological assessments, administration of anti-psychotic drugs and other patient outcome measures that would not necessarily be applicable in all care environments (Carpenter et al. 2006; Monette et al. 2008). While such Level 4b data could indeed be collected through use of a questionnaire, it would not always be geared towards the IPE learner, and would not lend itself to a single gold standard instrument.

Based on the insight gleaned through this review of the instruments currently in use in the literature, the reviewers have acknowledged the following:

1. No single questionnaire currently exists to comprehensively assess all IPE learner outcomes.

2. Few instruments or individual items consider behavioural change (Level 3), organizational change (Level 4a) or change in patient outcome (Level 4b) to any significant degree, relying instead on learner perceptions of the impact of IPE on these outcomes.

3. Items that might serve to provide objective assessment of higher levels may not always be generalizable to all practice environments, suggesting the infeasibility of a single gold standard instrument.

Given the above, future work should aim to create a toolkit to assist evaluators in the development of sound, meaningful evaluations of IPE and its ensuing impact on IPP. This could involve a combination of standard questionnaire items as well as subsets of optional items for use with certain education or practice environments. Such a toolkit could contribute to the generalizability of future IPE evaluation studies, as well as encouraging and facilitating the assessment of higher level outcomes.

Conclusion

The intent of this review was to document the published learner survey and questionnaire tools available to evaluate IPE learner outcomes, and to identify any gaps to be addressed in order to develop a gold standard tool. While a number of tools exist, none has been adopted as the gold standard, and none was comprehensive in its assessment of all levels of Kirkpatrick/Barr's IPE learner outcomes. We conclude that such a single comprehensive tool is not feasible due to the variety of educational and practice environments and to the accompanying need for individual outcome measures. We thus propose that future efforts be directed towards the development of an IPE evaluation questionnaire toolkit so that standardized questions can be utilized in a wide variety of contexts.

Declaration of interest: The authors report no conflicts of interest.