Quantitative methods in interprofessional education research: some critical reflections and ideas to improving rigor

Introduction

Although quantitative research on interprofessional education (IPE) is proliferating, the accelerating growth in this empirical work does not necessarily imply a stronger evidence base. The contrast between more empiricism and little-improved evidence is perplexing. On the one hand, quantitative research output has accelerated with little signs of slowing down. On the other hand, best-evidence syntheses have been limited to descriptive summaries, in part, due to the varietal methodologies across IPE studies (e.g. Reeves et al., 2016; Reeves, Palaganas, & Zierler, 2017). As Costanza (2015) noted recently, “in many ways, poor research methodology is the reason… research to date fails to show the impact of IPE” (p.35). Moreover, the methodological under-achievement of IPE may have been borne by early untended calls to focus IPE’s empiricism (Reeves, 2010).

This editorial provides an overview of the common quantitative study designs for IPE and addresses some implications for their analytic approaches. Subsequently, the editorial turns toward the prevailing measurement paradigm employed in IPE​—classical test theory (CTT). Specifically, the editorial discusses key limitations of CTT for IPE research and suggests how the adoption of modern psychometrics in IPE research – namely, item response theory (IRT) can remedy CTT’s limitations using substantive examples from IPE’s empirical literature.

Growth of quantitative research

Although the global increased-incidence of IPE is noteworthy, the concomitant use of varietal study methods limits the valuations and substantive impact of this form of education on practice and patients (e.g. Harath et al., 2017). Inconsistent methods partly owe to scarce guidelines, which have only recently become available for IPE researchers (e.g., Institute of Medicine, 2015; Reeves, Boet, Zierler, & Kitto, 2015). For quantitative research to sustain the development of IPE, stronger evidence may be achieved by offering more specific guidelines. Drawing on critical reflections from extant empirical literature, representative cases are presented below for use by IPE researchers.

Some common designs

In the Institute of Medicine (2015) report on IPE effectiveness, the conventional ‘gold-standard’ design of randomized-control trials (RCTs) is tempered to, “have limited utility for measuring the effects of IPE” (p.40). Because experimental designs are lesser-used in IPE studies, it is worth mentioning that the experiment’s “limitation” is likely due to classical inactive controls, which constricts the efficacy of IPE to absolute terms, such as ‘IPE’ vs. ‘no IPE’ (Brashers, Phillips, Malpass, & Owen, 2015). In contrast, active-controls (‘IPE comparators’) are suitable and appropriate for IPE evaluation in relative terms. For example, IPE experiments using ‘active controls’ may readily evaluate, comparative pedagogies (didactic lecture vs. problem-based learning), comparative measurements (single-Likert response vs. forced-choice format), or comparative contexts (classroom/acute-hospital vs. community-based settings).

For non-experimental (observational) designs, programs are typically evaluated by their usefulness for improving meaningful outcomes. For IPE researchers, different designs are optimal for different research questions and outcomes evaluated. Carpenter’s (1995) early seminal IPE study, for example, utilized a before-after design to evaluate changes in students’ interprofessional attitudes. In another example, as IPE institutionalization leads to more formalized curricula, the value-added of controlled before-after designs (CBAs) vis-à-vis ‘procedural confounds’² and ‘maturational threats’ can reasonably be expected to diminish for IPE research. Finally, the normative expectancies for IPE may approximate an ‘interrupted time-series’ (ITS) design with repeated IPE programs implemented to student populations over time. The changing proportion of IPE ‘elective’/‘compulsory’ offerings may inform the ‘usefulness’ of IPE reaction measures developed in the early stages of IPE institutionalization. More pragmatic for education administrators, the resource-intensiveness of ITS designs may limit their cost-benefit justification, particular for under-funded IPE programs. Of notable design-integration value, the ‘training transfer’ focus of long-term effectiveness of IPE (e.g., patient impact) is, recently, being directly abridged via community-based placements and regular IPE assessment (Kent, Martin, & Keating, 2016).

Modern measurement—practical benefits over classical approaches

Having overviewed some common IPE study designs, the next section below will turn to address some IPE measurement tools. The vast majority of existing IPE measures (e.g., Readiness for Interprofessional Learning Scale [RIPLS], Interprofessional Education Perceptions Scale [IEPS]) use CTT to construct and validate scales (Oates & Davidson, 2015). Modern measurement theories, such as IRT, offer several practical advantages over IPE’s current, CTT paradigm. We summarize a few, select advantages with illustrative examples to help substantiate researchers’ interpretation for the IPE field.

First, IRT allows for equating different measures that have been mapped onto a common metric (For example, within each level of Kirkpatrick’s modified-IPE outcomes framework).¹ Also, because IRT achieves greater reliability with fewer items, IPE studies may use a common subset of items (i.e., core items), which facilitates administration with shorter evaluations and improves student response rates with lower time-on-test (i.e., reduced response burden). Both of these features of IRT confer an advantage for strengthening IPE comparability across studies and sites that use different IPE measures.

Second, IRT enables item-level examination of measurement bias to strengthen IPE generalizability. Related to this more detailed, item-level scope of measurement bias, IRT allows for the use of IPE instruments beyond their original validation sample (e.g., student vs. practitioners). Removing this sample-dependent barrier to measurement selection gives greater access to researchers for optimizing content-appropriate IPE evaluations.

Third, in terms of understanding IPE change-scores, classical approaches severely limits interpretation, because well-documented baseline differences (e.g., Kerry, Heimberg, & Schmutz, 2017) of students and professions confounds meaningful assessments of change for learner outcomes. A salient example of CTT’s limitation for assessing change in IPE outcomes comes from Kashner and colleagues’ (2017) ten-year study of medical students’ preferences for IPE, where findings indicated a normative-increase in interprofessional preferences over time. Under a classical paradigm, this normative change would confer a decrease in before-after studies of IPE effectiveness. In contrast, IRT’s item-level examination would allow for revealing the – seemingly – ‘decreased effectiveness’ of IPE effectiveness as attributable to an item-level ceiling effect from normative changes of expectancies from IP training—Simply restated, when there’s less room to go up, there’s more chance of going down (Whereas CTT would suggest less IPE effectiveness, however, IRT would indicate a ceiling-bias and need for measurement revision).

Finally, related to IRT’s flexibility over baseline differences for IPE change-scores, IRT is also more flexible in combining scores from multiple sources (e.g., self-report/observer-rated), multiple item-formats (dichotomous ‘yes/no’ or checklists/Likert-type scales/forced-choice), and multiple content domains (reactions/attitudes/knowledge/behavior). This latter flexibility-advantage is imperative for IPE researchers wishing to evaluate the broader impact of IPE’s heterogeneous competency framework with a composite, global index.

Concluding comments

Analytically, very few IPE researchers have studied the identification of student individual-differences that impact on, for example, IPE receptivity, motivation for transfer, or candidacy for student-facilitator training. Identifying student characteristics can help IPE researchers strengthen the short-term effectiveness of IPE by targeting delivery to ‘in-need’ learners (Kerry et al., 2017). Accounting for relevant student characteristics would also help IPE researchers disentangle IPE’s long-term effectiveness, over the complex training pathway, for distal care practice and patient outcomes. Summarizing, IPE’s quantitative evidence may be strengthened by experimental comparisons of relative-IPE effectiveness (active-controls), a shift from classical to modern measurement (CTT→IRT), and more substantive analytics (individual differences). Meeting the complexities of IPE programs with such quantitative best-practice methodologies should, ultimately, sustain IPE’s future development and impact.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Notes

1. Kirkpatrick’s training evaluation framework, consisting of four criteria (trainee reactions, learning, transfer/behavior change, and organization results) was adapted for IPE by Barr and colleagues (2005). These authors expanded the original model to six outcome levels: (1) learner reactions; (2a) attitude/perception modification; (2b) knowledge/skill acquisition; (3) behavioral change; (4a) organizational change; and (4b) improvement to patient health/well-being.

2. Both terms refer to concepts known to weaken internal validity of non-experimental research study designs. Interested readers can learn by the following instructional link, http://www.creative-wisdom.com/teaching/WBI/threat.shtml.