Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative review)

Abstract

Background: Some important research questions in medical education and health services research need ‘mixed methods research’ (particularly synthesizing quantitative and qualitative findings). The approach is not new, but should be more explicitly reported.

Aim: The broad search question here, of a disjointed literature, was thus: What is mixed methods research – how should it relate to medical education research?, focused on explicit acknowledgement of ‘mixing’.

Methods: Literature searching focused on Web of Knowledge supplemented by other databases across disciplines.

Findings: Five main messages emerged: – Thinking quantitative and qualitative, not quantitative versus qualitative – Appreciating that mixed methods research blends different knowledge claims, enquiry strategies, and methods – Using a ‘horses for courses’ [whatever works] approach to the question, and clarifying the mix – Appreciating how medical education research competes with the ‘evidence-based’ movement, health services research, and the ‘RCT’ – Being more explicit about the role of mixed methods in medical education research, and the required expertise

Conclusion: Mixed methods research is valuable, yet the literature relevant to medical education is fragmented and poorly indexed. The required time, effort, expertise, and techniques deserve better recognition. More write-ups should explicitly discuss the ‘mixing’ (particularly of findings), rather than report separate components.

Introduction

As in health services, some medical education research questions (e.g. about new or complex initiatives and interactions (Schifferdecker & Reed 2009)) need ‘mixed methods research’:

[combining] quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study (Johnson & Onwuegbuzie 2004, p. 17)

investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry (Tashakkori & Creswell 2007, p. 4)

Despite some harmonization, ‘quantitative–qualitative’ turf wars flare up, featuring a Trojan horse of assumptions. Problems include quantitative institutional cultures and clashing cultures between medicine (e.g. ‘evidence-based medicine’) and education, and, within medical education, about notions of quality, evidence, population perspective, health policy, and heuristics (Sales & Schlaff 2010).

Medical education research should aim to improve students’ learning and their ultimate health impact, while reconciling eclectic views on how knowledge is created, discovered, learned, valued, justified, and verified, and challenging concrete ideas of ‘science’. Defining ‘science’ is contentious (Chalmers 1982; Regehr 2010), classifying medicine as possibly science, art, or both. Science crosses sundry social realities between objective and subjective assumptions about ontology (our existence), epistemology (our ideas about knowledge), human nature, and research methods (Burrell & Morgan 1979; Cohen & Manion 1994; Wilson 2000) (Box 1).

Box 1. Four dimensions distinguishing assumptions underlying ‘objective’ and ‘subjective’ approaches to social science

	‘subjective’		‘objective’
▪ Social reality is individually constructed from consciousness	Nominalism	ontology	Realism	▪ Social reality is external, imposed on consciousness
▪ Knowledge is personal, subjective, and unique	Anti-positivism	epistemology	Positivism	▪ Knowledge is hard, objective, tangible
▪ Humans initiate their own actions; as masters of free will	Voluntarism	human nature	Determinism	▪ Humans respond mechanically to environments, as marionettes
▪ The emphasis is on the particular and individual	Idiographic	methods	Nomothetic	▪ The emphasis is on discovery of general laws

Source: Burrell and Morgan (1979).

Note: Summaries at each end of box from Cohen and Manion (1994).

Wilson (2000) noted that medical schools promote objectivist assumptions (i.e. ‘enculturation’ of: patient–disease separation; simplistic cause-and-effect models; and keeping doctor–patient distance), with doctors viewing medicine mostly as a (‘biomedical’) ‘science’. To him, clinical practice needed both universal and existential approaches though, not just the ‘detached observer’ (gaining knowledge like a natural scientist). Cribb and Bignold (1999) noted that ‘it would be dangerously cavalier’ (p. 207) to dismiss this ‘detachment’ survival mechanism, but medical schools needed more reflexivity. Such debates in anthropology, physics and politics have allowed cultural relativity and more subjectivity (Wilson 2000).

The challenge ‘to look critically in our researches at the uniquely human elements in medical education’ (Mawardi 1967, p. 280) is longstanding. Defying quantitative tendencies to measure human behaviour by physical science goals and standards (used for test-tube chemicals or planets) (Buchanan 1992), Wilson (2000) urged a social constructivist approach of qualitative–quantitative equivalence. Constructivist theories (constructing understanding from our current knowledge and experiences) are contested, however, whether about learning (Colliver 1999; Giordan et al. 1999) or research (Colliver 1996a; Colliver 1999; Colliver 2002a; Jervis & Jervis 2005).

Beyond medical education, qualitative research is relatively popular (Roche 1991; Bergsjø 1999) in general practice (Boulton et al. 1996; Hoddinott & Pill 1997) and health services research (Pope & Mays 1995; Shortell 1999; Hoff & Witt 2000; Bradley et al. 2007; O’Cathain et al. 2007a; Lingard et al. 2008; O’Cathain 2009; O'Cathain et al., 2009). Qualitative research still struggles, however, for funding and mainstream medical acceptance (Cribb & Bignold 1999; Morse 2006; Dixon-Woods et al. 2007; Goguen et al. 2008; Sandelowski 2008; Pope & Mays 2009), undermined in ‘evidence’ hierarchies (Thorne 2009) and the hidden curriculum (Goguen et al. 2008). Challenges include the ‘evidence-based’ movement (Green & Britten 1998) (albeit evidence-based nursing appears more accommodating (Meadows-Oliver 2009; Thorne 2009; Broeder & Donze 2010)), outcomes research (Curry et al. 2009), and agreed standards, whether technical (Pope & Mays 2009) or ethical (e.g. potential distress, exploitation, distortion, and participant identification) (Richards & Schwartz 2002). Only 2% of original articles in seven journals (general medical, general practice and public health), 1991–1995, reported qualitative research (its very use sometimes meriting publication (Woolf 2006)), only 17% of these using mixed research (Boulton et al. 1996). Mixed methods went from 17% to 30% of commissioned health services research (Department of Health) between the early 1990s to 2002–2004 (O’Cathain et al. 2007a), and the number of Medline articles mentioning qualitative research increased progressively for the last two decades, up by 26% between 2001–2005 and 2006–2010 (first-quarter) (Ring et al. 2010).

‘Mixed methods’ suffer conflicting guidance on qualitative research standards (Mays & Pope 1995; Boulton et al. 1996; Hoddinott & Pill 1997; Chapple & Rogers 1998; Popay et al. 1998; Pope & Mays 1999; Stacy & Spencer 2000; Côté & Turgeon 2005). Stacy and Spencer (2000) understandably favoured being theoretically explicit (e.g. Abassi & Smith on behalf of British Medical Journal Education Group for Guidelines on Evaluation 1999) more than elusive researcher ‘independence’ (e.g. Harden et al. 1999, p. 557)), but medical education research reports rarely explore paradigmatic (theoretical perspective) assumptions (Mylopoulos & Regehr 2007; Bunniss & Kelly 2010) or substantive theory (Rees & Monrouxe 2010). The broad search question here, of a disjointed literature, was thus: What is mixed methods research - how should it relate to medical education research?, focused on explicit acknowledgement of ‘mixing’.

Searching literature databases (Haig & Dozier 2003a, 2003b) mostly used free-text in Web of Science (Science Citation Index Expanded, 1945-; Social Sciences Citation Index, 1956-; and Arts and Humanities Citation Index, 1975-) to August 2010 (Box 2), for - - -mixed method* and ‘(medical) education* (research)’- - -; medical education and ∼∼∼qualitative and quantitative and mixed)∼∼∼ or ∼∼∼evidence-based∼∼∼ or ∼∼∼research paradigm∼∼∼ (*=‘wildcard’). English language titles (±abstracts) yielded articles reviewing mixed methods theory or practice in (‘evidence-based’) health professional education (mostly undergraduate medical). Five key messages were apparent:

• Thinking quantitative and qualitative, not quantitative versus qualitative

• Appreciating that mixed methods research blends different knowledge claims, enquiry strategies, and methods

• Using a ‘horses for courses’ [whatever works] approach to the question, and clarifying the mix

• Appreciating how medical education research competes with the ‘evidence-based’ movement, health services research, and the ‘RCT’

• Being more explicit about the role of mixed methods in medical education research, and the required expertise

Table

Practice points

Write-ups should report the ‘mixing’, as appropriate, rather than reporting qualitative–quantitative components separately, and it remains helpful to acknowledge the pragmatism paradigm, given that this research is primarily ‘research question’-driven (modified by feasibility and context), rather than theory-driven per se. Write-ups should explicitly highlight what has been mixed, implicitly acknowledging the required rigour, expertise, time, and effort. Write-ups should explicitly discuss ‘mixing’ findings particularly.

Box 2. Literature search strategy: ‘What is mixed methods research, and how does it relate to medical education research?’

▪ Mainly used free-text in Web of Science (Science Citation Index Expanded, 1945-; Social Sciences Citation Index, 1956-; Arts and Humanities Citation Index, 1975-) to August 2010, with supplementary searches.

▪ Used: Medline, 1966-; AMED (Allied and Complementary Medicine), 1985-; British Nursing Index (BNI), 1985-; CINAHL (Cumulative Index to Nursing and Allied Health Literature), 1982-; International Bibliography of the Social Sciences and 1951-; ERIC (Educational Resources Information Center), 1966-; and PsychINFO, 1971-

▪ Combined the search terms: —mixed method* and ‘(medical) education* (research)’- - -; medical education and ∼∼∼qualitative and quantitative and mixed)∼∼∼ or ∼∼∼evidence-based∼∼∼ or ∼∼∼research paradigm∼∼∼ (*=‘wildcard’ character(s))

▪ Checked English Language titles (±abstracts) for articles reviewing mixed methods theory or practice (including relevance to being ‘evidence-based’) and examples in health care education (mostly undergraduate medical) or allied settings

▪ Searched: the University library catalogue for ‘mixed methods’ books; and used ancestry searching (from reference-lists), ad hoc ‘finds’, personal collection and January 2004–August 2010 handsearching of journals: Medical Education, Medical Teacher, Advances in Health Sciences Education, Academic Medicine, and Journal of Mixed Methods Research

Source: Haig and Dozier (2003a, 2003b).

Thinking quantitative and qualitative, not quantitative versus qualitative

The zealous ‘quantitative versus qualitative’ research debate appears inescapable, yet somewhat futile (Niglas 1999), steeped in military rhetoric (‘a phony war’ (Bergsjø 1999, p. 561) and ‘more a battlefield than a field of inquiry’ (Norman 1998, p. 77), prompting ‘Why can’t we all just get along?’ (Onwuegbuzie 2000, p. 11)). In education, Onwuegbuzie (2000) traced this debate to the late 1800s, when logical positivism underpinned ‘science’, seeking ‘verifiability’ via systematic ‘hard’ data collection of objective evidence (with probabilistic and inferential analysis) to explain, predict, and control phenomena. Psychosocial researchers soon backed interpretivism, such as hermeneutics (Onwuegbuzie 2000) – seeking meaning from participants (rather than external ‘truth’). This contrasted with refinements of positivism in quantitative research, by researching in everyday settings with minimal interference (and knowledge a matter of interpretation – interpretivism), yet still using scientific method (naturalism) (Pope & Mays 1999). The main schism remained (Onwuegbuzie 2000) (Box 3).

Box 3. Four paradigms for considering knowledge claims and theory, the six key strands of the pragmatism continuum, and the ‘schism’ it spans

– ‘postpositivism: determination, reductionism, empirical observation and measurement, theory verification …

– constructivism: understanding, multiple participant meanings, social and historical construction, theory generation …

– advocacy/participatory: political, empowerment issue-orientated, collaborative, change-orientated …

– pragmatism: consequences of actions, problem-centred, pluralistic, real-world practice orientated’ (Creswell 2003, p. 6)

Six key strands of the ‘pragmatism’ continuum (Tashakkori & Teddlie 1998):

– Methods: use both qualitative and quantitative

– Logic: use both inductive and deductive

– Epistemology (knower↔known relationship): accept subjectivity and objectivity (‘epistemological relativism’)

– Axiology (the role of values): accept that values influence interpretation

– Ontology (nature of reality): accept ‘external reality’, but the best explanations for outcomes

– Causation: accept that causal relationships are possible, yet elusive

The schism that pragmatism spans:

‘ … realism versus idealism, foundational versus antifoundational, objective versus subjective, hard versus soft, scientists versus critics, personal versus impersonal, deductive reasoning versus inductive reasoning, rigor versus intuition, generalization versus uniqueness, logistic versus dialectic, rationalism versus naturalism, reductionism versus holistic, causal versus acausal, macro versus micro, correspondence versus coherence, quantifiers versus describers, and numbers versus words’. (Onwuegbuzie 2000, p. 13)

Purists claim paradigm superiority, like a moral crusade (Onwuegbuzie 2000), pledging allegiance (Johnson & Onwuegbuzie 2004). The debate:

… has tended to obfuscate rather than to clarify, to stereotype rather than to enlighten, and to divide rather than to unite educational researchers. (Onwuegbuzie 2000, p. 10)

Paradigms become barriers (Niglas 1999), yet both approaches have strengths and weaknesses, and underpin social research. Johnson and Onwuegbuzie (2004) and Punch (1998) noted quantitative–qualitative similarities in data type and handling. From nursing, Goodwin and Goodwin (1984) dismissed myths about certain methods being exclusive for certain paradigms (despite others defining qualitative research thus (Boulton et al. 1996)) and about qualitative research being always or exclusively unobtrusive, naturalistic, and subjective. Like Goodwin and Goodwin (1984), Morse (1999a) dismissed the supposed irrelevance of validity and reliability in qualitative research (and also debunked the non-generalizability myth (Morse 1999b)). Newman (2000) dismissed another myth:

There is a frequently held misconception that quantitative research uses numbers and qualitative research is narrative. This is a misleading simplification…it is not the technique that makes something quantitative or qualitative, but it is the intent of its uses. Is it testing hypotheses or is it helping to develop hypotheses or describe the data (Newman 2000, pp. 4–5)

For Punch (1998) it was ‘not inevitable, or essential, that we organize our empirical data as numbers’ (p. 58), but both approaches could induce or test theory.

The pragmatism paradigm (in the sense of a ‘world-view’ or approach) (Bergman 2010) of ‘mixed methods’ reduces tough choices between methods, logic, or epistemology (Tashakkori & Teddlie 1998), reaching Onwuegbuzie's (2000) ‘epistemological ecumenism’ (p. 11). In psychosocial sciences, Tashakkori and Teddlie (1998) cast ‘pragmatism’ as pacifist in ‘the paradigm wars’ (usually positivism versus constructivism), contrasting its American roots (e.g. Dewey (Hallet 1997), Rorty (2000)) with European caution over debunking metaphysical truth (to ‘what works’). An alternative (post-positivist) paradigm, ‘critical realism’ (Colliver 1996b; McEvoy & Richards 2006) accepts an external reality with multiple (albeit fallible) outlooks to seek sense (via observation, measurement, perception, or cognition) (Trochim 2006), but strongly rejects relativism, while pragmatism seems agnostic on it (Proctor 2004).

In summary, mixing uses the pragmatism paradigm, an inclusive ‘what works’ approach to ‘truth’. This reconciles assumptions about social reality, avoiding futile, ‘either–or’, qualitative–quantitative polemics. The key ideas show agnosticism rather than bland compromise.

Appreciating that mixed methods research blends different knowledge claims, enquiry strategies, and methods

From educational psychology, Creswell's (2003) three questions (from Crotty's (1998) model) distinguished qualitative, quantitative, and mixed methods research.

What knowledge claims and theory (the paradigm)?

Creswell (2003) includes the pragmatism paradigm (besides postpositivism, constructivism, and advocacy/participatory (Box 3)), i.e.:

• No specific schemes of philosophy and reality; free choice of methods; indifference to ‘qualitative or quantitative’; and belief in ‘qualitative and quantitative’, truth being what works at the time

• Focusing on the purpose of research; appreciating social, historical, and political context; unperturbed by disputed reality or disputed laws of nature.

To Cherryholmes (1992, p. 16), ‘pragmatists’ searching for reality was hopeless: ‘Even if we came upon a True account of what is ‘real’, we would be at a loss to recognize it as True’, whereas ‘scientific realists’ romantically seek an objective independent reality. Onwuegbuzie (2000) disputed purists’ self-defeating assumptions:

Qualitative: ‘All truth is relative’ would be true only relatively. Accepting ‘There are multiple realities’ should allow the ‘quantitative’ version of reality. (Universal assertions that there are no universals are indeed ironic (Norman 1998; Colliver 2002a).)

Quantitative: ‘The verifiability principle’ (assertions are only meaningful if verifiable) is not empirical or logical.

What enquiry strategy (associated traditions of enquiry)?

Classifications of qualitative, quantitative, and mixed methods research differ considerably. Social research literature tends to see ‘experiments’ or ‘surveys’ (Creswell 2003) rather than, for example, the differentiated clinicoepidemiological hierarchy of quantitative study design (i.e. case report, case series (clinical or population), cross-sectional study, case-control study, cohort study, and randomized controlled trial (Bhopal 2008)). Quantitative research tends not to follow ‘traditions’ explicitly per se.

Qualitative research strategies explore, describe, or generate theory, especially for uncertain and ‘immature’ concepts (Morse 1991); sensitive and socially dependent concepts (Roche 1991); and complex human intentions and motivations (Harris 2003). This generally ‘case-orientated’ (not ‘variable-orientated’) (Punch 1998) research favours open-ended questions, unstructured approaches, and highlighting differences rather than averaging responses (Roche 1991). Classifications abound – for example, Creswell (1998) outlined five main traditions (from about 20): biography/narrative, phenomenology (underused in medical education), grounded theory, ethnography, and case studies, while Grbich (1999) noted field-, action-, or library-based approaches in health research. Classifications of mixed methods research, avoiding ‘mixed up methods’ (Tashakkori & Teddlie 1998), include the simplified Teddlie and Tashakkori (2006) typology of ‘mixed’ versus ‘quasi-mixed’ (the latter being without substantial integration of findings and inferences).

What data collection and analysis methods?

Competing advice about ‘doing’ qualitative research (Chapple & Rogers 1998) is off-putting, as are much criticized recipe-like checklists (Barbour 2001), and ‘feigning “immaculate perception”’ (Wolcott 1992, p. 43) (e.g. claiming exclusivity for trained social scientists). ‘Pragmatism’ is not precious about approach. Educational researchers have promoted qualitative–quantitative ‘mixing’ (Punch 1998; Tashakkori & Teddlie 1998; Creswell 2003; Elliott 2004; Johnson & Onwuegbuzie 2004; Raudenbush 2005; Demerath 2006; Miller & Fredericks 2006), e.g. ‘ … we should be shamelessly eclectic in our use of methods’ (Rossman & Wilson 1991, p. 17). Such mixing occupies a continuum (Newman 2000) though, as do methods, logic, epistemology, axiology, ontology, and causation in pragmatism (Tashakkori & Teddlie 1998) (Box 3).

In summary, mixing methods (a ‘movement’ in only its third decade (Creswell & Garrett 2008)) blends eclectic views of knowledge, traditions of enquiry, methods, and results; stays practice-orientated; and uses ‘what works’, not elitist stances. Classifications do not necessarily clarify ‘the mix’ though.

Using a ‘horses for courses’ [whatever works] approach to the question, and clarifying the mix

Rossman and Wilson (1985) noted three stances about qualitative–quantitative mixing:

purism: they cannot be combined and one is favoured

situationalism: both are valuable, maybe in one study, but only in their place

pragmatism: both are valuable, especially combined, whether in study design, data collection, or analysis.

Their rationale for ‘pragmatic’ combination could be: corroboration (true triangulation), elaboration (‘If we think of social phenomena as gems, elaboration designs are intended to illuminate different facets of the phenomenon of interest’ (Rossman & Wilson 1991, p. 2)), initiation, and/or development (Box 4), and Onwuegbuzie et al. (2010) highlighted instrument development and construct validation.

Box 4. The rationale for mixing quantitative and qualitative research

▪ Corroboration: of convergent findings, i.e. true ‘triangulation’ (Morse's (1991) concurrent validation of the same issue from different sources):

– Triangulation is a much misused term (Greene et al. 1989), often to claim rigour (Barbour 1998). Tashakkori and Teddlie (1998) noted Denzin's (1978) four basic types: data, investigator, theory, and methodological; and that ‘true triangulation’ was the ‘intellectual wedge that eventually broke the methodological hegemony of the monomethod purists’ (p. 41).

▪ Elaboration: enriching findings of one approach with the other (Greene et al's (1989) ‘complementarity’):

– Morse (1991) described simultaneous versus sequential methodological triangulation (depending on when mixing occurred), but contrasted simultaneous triangulation (of different aspects; complementary methods, ‘elaboration’) with true triangulation (concurrent validation: i.e. same concept, different methods; as confirmed by others (Kadushin et al. 2008)).

▪ Initiation: rethinking ideas, suggesting further work and seeking the provocative

▪ Development: one orientation informing the other (was a later addition)

Greene et al. (1989) used a fifth category, ‘expansion’ (of an evaluation project, e.g. exploring both process and outcome).

Reviewing 57 education evaluations, Greene et al. (1989) found that complementarity or expansion formed four-fifths of primary aims (and one-half of total aims), but combined data analysis was rare.

After Rossman and Wilson (1985).

On the mixing continuum, Punch (1998) meant: adding, interweaving, integrating, or linking methods, data, and/or findings with increasing complexity. Creswell (2003) meant: implementing ‘mixing’ simultaneously, sequentially, and/or transformatively; showing priority to qualitative, quantitative, or both equally, integrating data collection, analysis, and/or interpretation, and being theoretically explicit or implicit. Visually representing key ‘mixing’ decisions might help (Ivankova et al. 2006).

Some published examples (O’Sullivan 2010) do not report or critique details of mixing data collection via a single questionnaire, for example, yet this can give meaningful mixing (e.g. Johnson and Onwuegbuzie's (2004) ‘within-stage mixed model design’):

For example, in data collection, this ‘mixing’ might involve combining open-ended questions on a survey with closed-ended questions on the survey. Mixing at the stage of data analysis and interpretation might involve transforming qualitative themes or codes into quantitative numbers and comparing that information with quantitative results in an ‘interpretation’ study. (Creswell 2003, p. 212).

Combining approaches can mean simply incorporating open-ended questions in a fixed-choice self-completion questionnaire, or systematically collecting quantitative information (such as age or length of an experience) during interviews or focus groups. (Barbour 1999, p. 40)

Tashakkori and Teddlie (1998) related pragmatism to the ‘scientific method’ research cycle of inferences, but with induction and deduction more apparent (possibly simultaneous):

The research question predominates in mixed methods in mainstream education research, and, for example, health services research (Barbour 1999), nursing research since the 1990s (Morse 1991; Carr 1994; Sandelowski 2000), and more recently with health sciences research (Andrew & Halcomb 2009), and engineering education (Leydens et al. 2004; Borrego et al. 2009). This ‘horses for courses’ eclecticism of medical education research is increasingly acknowledged (Abassi & Smith on behalf of British Medical Journal Education Group for Guidelines on Evaluation 1999; Bligh & Anderson 2000), but needs better quality research questions (Schuwirth & van der Vleuten 2004; Shea et al. 2004). If science essentially seeks better story-telling about how the world works, then: ‘while the particular rules of the story may differ, just as the rules of a sonnet differ from a limerick, good stories are independent of the form’ (Norman 1998, p. 79).

Tashakkori and Teddlie (1998) highlighted the required ‘dictatorship of the research question (not the paradigm or method)’:

pragmatists consider the research question to be more important than either the method they use or the worldview that is supposed to underlie the method…For most researchers committed to the thorough study of a research problem, method is secondary to the research question itself, and the underlying worldview hardly enters the picture, except in the most abstract sense. (Tashakkori & Teddlie 1998, p. 21)

Oppenheim advised likewise:

… It would be more helpful to suggest that choosing the best design or the best method is a matter of appropriateness. No single approach is always or necessarily superior; it all depends on what we need to find out and on the type of question to which we seek an answer. (Oppenheim 1992, p. 12)

Rigid ‘quantitative versus qualitative’ stances then appear facile. Punch (1998) considered that approach should also reflect context, current literature, feasibility, potential cost-benefit, and personal expertise/experience, while Creswell (2003) also highlighted the researcher's experience and potential audience.

Challenges remain though. Buchanan (1992) argued that quantitative research dominated social science because of: ‘scientific method’ being successful in understanding the natural world; comforting ‘certainties’ in ‘hard’ science; government and funding support; wanting the ‘perfect’ experiment; rejecting subjectivity; and unfamiliarity with qualitative research (goals, standards, and assumptions). He favoured qualitative research in any mixing, yet was sceptical about mixing because:

• Quantitative research disregards outliers, whereas qualitative research highlights the singular response (the exception), because that responder might be more perceptive or articulate in raising this ‘important’ issue.

• Qualitative research aims for logical, not probabilistic, generalization (also Popay et al. 1998). Logical inference is, however, problematic when quantifying qualitative data by dichotomizing complex answers about opinions, which do not necessarily relate one-to-one, linearly, with behaviour.

• Themes/‘ideal types’ (abstract descriptions of typical constructs) emerge piecemeal but convincingly across an interview, yet defy ‘scoring’.

• Quantitative research does not help with all-pervasive (universal) themes.

Howe (2004) promoted ‘mixed methods interpretivism’ over two educational research developments he perceived would undermine qualitative research by focusing on ‘cause’, ‘effectiveness’ and randomization:

‘neoclassical experimentalism’: … even more restrictive designs

‘mixed methods experimentalism’: … tokenistic addition of qualitative components

In summary, the mix in mixed methods research can vary by type, extent, and intention, so researchers should be clear what best answers their research question. Factors such as context, evidence-base, and feasibility (including cost-benefit and personal expertise) also affect what/how to mix. Sometimes this involves true ‘triangulation’ (a term best used for exploring the same issue in different ways). One instrument might suffice, e.g. a questionnaire, but mixing should be meaningful, not tokenistic.

Appreciating how medical education research competes with the ‘evidence-based’ movement, health services research, and the ‘RCT’

Medical education research and theory are critiqued harshly (Box 5) through a lens of ‘evidence-based medicine’ (Leung & Johnston 2006). They seek ‘evidence-based’ credibility (Wartman 1994; Albanese et al. 1998), struggling since their structured foundations in the late 1950s/early 1960s (McGuire 1996) (and informal studies of medical students’ personality and intelligence in the early 20th century (Levine et al. 1974)). Some detractors (Wartman 1994) expected too much applied output from the ‘formative’ stages of this research (Norman & Schmidt 1999), which apparently trails behind medical domains of similar vintage such as clinical epidemiology. Problems include:

• Inadequate questions, designs, and samples (Dauphinee 1996; McGuire 1996) being neither truly basic nor clinical science (Friedman 1996)

• Attracting ‘me-too research’ (Norman 2006), ‘saying little more than that the students liked the innovation’ (Abassi & Smith on behalf of British Medical Journal Education Group for Guidelines on Evaluation 1999, p. 1265)

• Historical underfunding (Wartman 1994; Albanese et al. 1998), weak institutional support (Lovejoy & Armstrong 1996; Albanese et al. 1998), vaguer productivity measures than for health care (research), and the effect of some health care systems (e.g. United States managed care undermining innovation, funding, and research for medical education) (Albanese et al. 1998).

Box 5. Contrasting contemporary views of medical education research

– ’It is very difficult to undertake meaningful research in education. The variables are too diffuse and difficult to identify. Very often they are not easy to measure. Other factors often contaminate the relationship between an educational event and its eventual outcome. Sometimes, particular outcomes are not easy to specify, nor are the timescales in which we might expect to see an effect or to see an effect last’. (Harden et al. 1999, p. 559)

– ‘ … High-stakes educational decisions regarding admission, promotion and accreditation may not be as life-threatening as life-and-death clinical decisions, but some of the consequences of these decisions certainly are irreversible and can affect individuals’ careers, quality (if not quantity) of life, job performance and productivity’. (Wolf 2000, p. 251)

– ‘Perhaps the highest level of professionalism in being a teacher is to contribute to the accumulation of evidence, to ‘problematize’ one's own educational situation, to start investigating and to engage in educational research’. (van der Vleuten et al. 2000, p. 249)

– [In response to Colliver's (2000) harsh critique of the educational evidence and theory underpinning problem-based learning (PBL)] ‘Does this mean … that all efforts at educational research are fruitless? Not at all. But we must take a cue from the natural sciences and move away from blind allegiance to the canons of sound methodology (randomization and all that) to recognition and support for research programmes, whose intent is to create an environment where ideas are shepherded from the basic science laboratory to the application setting’. (Norman & Schmidt 2000, p. 726)

– ‘Cognitive science is often said to be the basic science of medical education, and cognitive theory is routinely cited as a justification for educational practice. … a critical look at the theory [underpinning PBL] shows it is mostly metaphor, not rigorous, tested, confirmed scientific theory’. (Colliver 2002b, pp. 1217, 1220)

– ‘ … if the search for ‘replicable educational programmes’ is an implausible goal, if there are no meaningfully generalizable solutions in health professional education, then what is the purpose and value of science and scientific discourse in the health professional education field? [ … ] Education research is not rocket science, which is built on a structured, linear system with a straightforward set of factors which we can stick into a well-articulated formula to predict a clearly defined outcome. Rather, if we must make analogies to the physical sciences, we might do better to look to quantum mechanics and chaos theory. Such analogies will lead us away from the search for proofs of simple generalizable solutions to our collective problems, and towards the generation of rich understandings of the complex environments in which our collective problems are uniquely embedded’. (Regehr 2010, pp. 37–38)

Norman and Schmidt (1999) urged more basic theory-building cognitive psychology research, and McGuire (1996) considered that research should: redefine medical education goals against a robust concept of the competent doctor; help design relevant curricula (in an evidence-based way (Taylor 2010)); and evaluate cost-effectiveness. Despite cognitive science giving coherent insights (Reese 1998; Mayer 2010), Colliver criticized it (and its embodiment in problem-based learning (PBL) (Dolmans & Schmidt 1996)) (Colliver 2000), rebuffing excuses for its infancy (Norman & Schmidt 1999). For Colliver (2002b, p. 1220), ‘educational innovations and practice claims are at best conjecture, not evidence-based science’, and the major reviews of PBL effectiveness (Albanese & Mitchell 1993; Berkson 1993; Vernon & Blake 1993), for example, were unpersuasive, irrespective of scant formal evidence for ‘conventional’ education. For Colliver (2002b), educational theory was metaphor, ‘not rigorous, tested, confirmed scientific theory’ (p. 1217).

The relative weakness of such research extends beyond deficient expertise. Murray (2002) listed barriers such as: complex educational interventions; problems randomly allocating interventions; elusive outcomes and measurement tools; underfunding; and especially clinicians’ disinclination. Petersen (1999) also urged medical educationalists to use accessible terminology and improve research designs, while awaiting graduates of innovative curricula to improve attitudes. Van Der Vleuten et al. (2000) noted how university staff perceived research or professional practice markedly differently from education, where, ‘ … any challenge to one's convictions is an actual challenge to one's professional integrity’ (p. 246). They argued that using tradition and intuition had led to such flawed notions as: ‘teaching is learning’, ‘the more we teach the more students learn’, ‘competence consists of distinct competences’ (confirmed by the non-existence of content-independent ‘problem-solving’ skills), and ‘the curriculum [rather than assessment] dictates learning’. Petersen noted:

… the same professional standards are not so commonly applied. All doctors have been successful medical students, and it seems easy to assume that this alone qualifies them to educate others. Few surgeons would claim that surviving a surgical procedure qualifies a patient to perform it on another (Petersen 1999, p. 1223)

Medical education research has become more robust (Baernstein et al. 2007) (and, arguably, sufficiently robust evidence exists to reassure and act (Norman 2000)), but how it should develop (Shea 2001; Prideaux 2002a), focus, and learn from other fields seems uncertain. Directions of influence with other fields are debatable. Cochrane Collaboration ‘evidence-based medicine’ has influenced the international Best Evidence Medical Education (BEME) collaboration in synthesizing evidence (Wolf et al. 2001) – and evaluating against ‘QUESTS’ dimensions (quality, utility, extent, strength, target and the setting) places the best available evidence on an ‘evidence-based↔opinion-based’ continuum (Harden et al. 1999). Wolf et al. (2001) noted that the Cochrane Collaboration revisited a term – ‘meta-analysis’ – from an American Educational Research Association presidential address in the mid-1970s. Mainstream education research thus remains influential (Lloyd 1991), with Harris (2003) attributing this and social science influences to researchers’ close links with educational psychology and the ‘biomedical’ research culture of medical schools – but health services research is influential too.

Prystowsky and Bordage (2001) applied a health care outcomes research framework for their content analysis of medical education research, and found product cost and quality of medical education to be under-researched. Shea (2001) disagreed with their applying health services research examples, because:

• The ‘primary customer’ is the learner not the patient.

• Dilution makes it nearly impossible to show learners’ outcomes affecting patients’ outcomes. (McGuire (1996) lamented the ‘inexcusable shortage of outcomes research’ (p. S125), wanting impact measured in health care currency, but this still seems overambitious.)

• Medical education often changes before strong study designs are possible (Taylor 2010).

• Cost analysis may often be supplementary, which the authors presumably omitted in studying one main focus per article.

Conversely, Murray (2002) valued health services research lessons, where evaluating similarly complex interventions required combined qualitative and quantitative approaches, but omitted the thorny technical barrier of ‘mixing’, and its perceived ‘sudden faddishness’ (Morse 2005, p. 583).

‘Evidence-based education’ (Davies 1999) remains aspirational. Wolf (2000) considered that evidence-based medical education would probably develop similarly to evidence-based medicine, where the ‘critical appraisal’ step (then ‘finding best evidence’) had made the most progress (▸asking relevant answerable questions; ▸finding best evidence efficiently, ▸critically appraising it; ▸using expertise to adapt and apply evidence; ▸evaluating impact). Better research questions are needed (Schuwirth & van der Vleuten 2004; Shea et al. 2004), maybe to altruistic goals (Sestini 2010). The first of Wolf's (2000) 10 lessons from evidence-based medicine – synthesizing evidence is usually more complex and complicated than anticipated – applies particularly when synthesizing non-RCT evidence (from qualitative (Thorne 2009; Broeder & Donze 2010) and/or quantitative approaches). Mining gems from meaningful but messy medical education research about ‘colourful’ phenomena needs a robust yet inclusive research constitution, and a wider world-view than just the RCT or any other potentially monochromatic mindset. Better research will require ‘elaboration of the messy parts of our efforts to intervene’ (Regehr 2010, p. 38) in education.

RCTs fit poorly into social research (Cook & Shadish 1994), although mixing with qualitative research helps (Bradley et al. 2005; Moffatt et al. 2006), and a sea-change to medical educational epidemiology (Carney et al. 2004) or single-case experimental designs (Bryson-Brockmann & Roll 1996), for example, would not answer many quantitatively orientated questions. Exploring BEME assumptions, Norman (2000) agreed that education research eludes universal standards, but not because it is ‘soft’, noting that many clinical questions also defy the ‘universal approach’ of RCTs (and thus, noted Gillett (2004), ‘[their] current fundamentalism’ (p. 730) and ‘positivist conceptions of argument and investigation … of evidence-based medicine’ (p. 732)). In selected clinical treatment areas, Benson and Hartz (2004) found that observational study evidence was reasonably solid compared with RCTs. Cohort studies are quite robust (Goldacre 2001), and Concato et al. (2004) found that allegedly inflated ‘treatment effects’ were unfounded with contemporary controls. The ‘perfect study’ might well be useless (Lloyd 1991).

Norman (2000) noted BEME aptly included different epistemologies but, by seeking uniform quality assurance, assumed unidimensionality. Moreover, he noted:

• Unlike drug doses, educational interventions are rarely standard, reducing transferability from RCTs.

• Mostly, one ‘world-view’ judges the strength and breadth of evidence, yet one good study can be enough, small p-values do not necessarily mean large effect sizes, and many worthy research questions do not translate into effect sizes anyway (and are not clearcut quantitative↔qualitative issues).

• BEME may well underestimate the generalizability of medical education evidence, as well-established examples exist.

Everyday education defies randomization, blinding, and controlled interventions, and lacks good outcome measures (Norman & Schmidt 2000; Prideaux 2002b) though, and ‘trials of curriculum level interventions … are … a waste of time and resources’ (Norman & Schmidt 2000; disputed by Colliver 2004). The PBL evidence-base, for example, suffers: confounders; small, context-bound, single-site studies; varied PBL definitions in disparate settings; and unsuitable conditions for randomized controlled trials (Finucane et al. 1998).

In summary, like health services research, medical education research suffers expectations from evidence-based medicine and the ‘RCT’ mirage, despite different: ‘customers’, dilution effects, timescales of change, and cost. Complex, context-specific questions and settings bedevil medical education research, and require expertise (such as in ‘messy’ mixed methods approaches) that clinical, academic, and institutional cultures undervalue and underfund. Medical education research could usefully focus more on: theory-building cognitive psychology (Norman 2004); illuminating the competent doctor, curriculum design (overcoming a ‘know-do’ gap (Levinson 2010)), outcomes and cost; challenging long-held assumptions that favour teaching over learning; improving both primary research and research synthesis (while accepting ‘imperfect’ designs); and embracing eclecticism of epistemology and enquiry.

Being more explicit about the role of mixed methods in medical education research, and the required expertise

Further quandaries for mixed methods medical education research specifically include the time, effort, and expertise required (Morse 1991; Creswell 2003). Like health services research (O’Cathain 2009), the approach itself is insufficiently discussed, e.g. appropriate sampling (Teddlie & Yu 2007) and practicalities (Schifferdecker & Reed 2009). Besides, the preferences of the researchers, disciplines, funders, and publishers, barriers to mixing (Bryman 2007) include insufficient practical guidance (Schifferdecker & Reed 2009), insufficiently detailed write-ups of what was integrated and how (O’Cathain et al. 2007b, 2008, 2010; Bryman 2007; Alise & Teddlie 2010; O'Cathain et al. 2009), and dissemination meeting inertia (Wilson 2010)

Teddlie and Tashakkori (2009) cast their prototypical mixed methods researcher, ‘Professor Eclectica’, in public health. Prideaux (2002a, p. 502) highlighted the ‘sophistication in thinking and understanding’ required to research medical education across research traditions. Assimilating diverse types of evidence (Leung 2002) and the identity crisis of spanning disciplines (and being disowned in-between) are barriers. Prideaux reinforced the ‘ … “virtue” in embracing “eclecticism” … ’ (2002a, p. 502), whether by one researcher spanning research traditions or many researchers collaborating from different backgrounds. The ‘lone researcher’ needs diverse skills and different logical principles (Mason 1994) to undertake time-consuming work and negotiate purism. Ironically, some researchers might shun mixing assumptions across the main approaches, yet happily mix methods, with very different assumptions, across traditions within qualitative research (Barbour 1998).

Specific, labelled, examples of mixed methods research in undergraduate medical education should be more prominent, but include, for example, exploring medical students’ early patient contact (Howe et al. 2007) and learning in the operating theatre (Lyon 2003) (and academic surgeons as educators in theatre and clinic (Cox & Swanson 2002)). Others have explored small-groupwork dynamics (MacPherson et al. 2001), programme evaluation (Gerrity & Mahaffy 1998), and interprofessional learning (Bradley et al. 2009). Frye et al. (1993) commended a mixed approach to explore medical students’ complex learning environment in a problem-based parallel track while rotating through clerkships, and Maudsley et al. (2007) used a questionnaire-based mixed methods approach to explore students’ perceptions of a good doctor and of learning in a problem-based curriculum (2008).

Barbour (1999) and Creswell (2003) recognized mixed methods potential in the questionnaire, but it remains rather misused and maligned, and lists of ‘bona fide’ qualitative research data collection methods (Boulton et al. 1996) usually omit it. Bergsjø (1999) recognized its role in qualitative research, although as ‘the most programmatic approach’ (p. 560). Frye et al. (1993, p. 46) reported mixing ‘five qualitative data collection methods’, including 5-min questionnaires comprising open-ended questions only. They considered it unobtrusive and efficient at giving complementary insights: ‘No single method captures the “big picture”’ (p. 59).

Compared with the revered semistructured interview (Cicco-Bloom & Crabtree 2006), however, questionnaire weaknesses are often highlighted, whether from social desirability bias, acquiescence bias, rigidity, dogmatism or authoritarianism (Oppenheim 1992). Poor questionnaire design (or application) adds to ‘a bit of a study’ medical education research culture.

In summary, mixed methods research is valuable in medical education, but the required time, effort, expertise (in ‘messiness’), and mixing techniques are seldom explicitly discussed. A carefully designed ‘questionnaire’ has much potential, but sloppy use of ‘a bit of a questionnaire’ for ‘a bit of a study’ undermines medical education research.

Comment

Although as Eva (2008) noted, interpreting and integrating highly varied education literature will tend towards quirkiness, this critical narrative review highlights some recurring messages, emerging over the last two decades, about mixed methods research, relevant to medical education. Medical education research struggles self-consciously for credibility against:

• RCT-driven evidence-based medicine and health services research;

• a jumble of philosophies, concepts, assumptions and criticisms; and

• inadequate study designs pursuing unrealistic questions and expectations.

Using the pragmatism world-view, mixed methods is research question-driven, conciliatory and underpins much robust research in education generally. The ‘mixing’ varies by type, extent and intention (blending eclectic views of knowledge, traditions of enquiry, methods and findings). Mixing requires expertise and resilience, amongst clashing expectations and cultures with other medical research.

Current literature about mixed methods focuses on the theory, techniques, nature, use and politics (Creswell 2009), but in medical education this is fragmented and poorly indexed. More write-ups should explicitly discuss the ‘mixing’ (Alise & Teddlie 2010; O’Cathain et al. 2010) (particularly of findings), rather than reporting qualitative–quantitative components separately. Mastering the mixing ‘trade’ involves harnessing the cognitive dissonance (Norman 1998; Colliver 2002a) and complementary strengths (Johnson & Onwuegbuzie 2004; Niaz 2008), while challenging the ‘incompatibility thesis’ dogma (forbidding mixture of paradigms and methods, Howe 1988) and ‘Jack of all trades, master of none’ criticisms. As Norman (2008) noted, the ‘RCT’ and the ‘systematic review with meta-analysis’ (Norman & Eva 2008) are usually problematic here, and ‘The time is long overdue to abandon the worship of the false God of the RCT’ (p. 388). Mixing methods is not new, and seems increasingly relevant to medical education (Durning et al. 2010) – being robust and explicit about its theory and practice is overdue.

Ethical approval

Not applicable.

Funding: Nil

Acknowledgements

Thanks to my thesis supervisors, Dr Lyn Williams and Dr David Taylor, all that time ago, who commented constructively on my original review that I subsequently developed into this article.

Declaration of interest: The author reports no conflicts of interest. The author alone is responsible for the content and writing of this article.