Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative review)

Figures & data

Box 1. Four dimensions distinguishing assumptions underlying ‘objective’ and ‘subjective’ approaches to social science

	‘subjective’		‘objective’
▪ Social reality is individually constructed from consciousness	Nominalism	ontology	Realism	▪ Social reality is external, imposed on consciousness
▪ Knowledge is personal, subjective, and unique	Anti-positivism	epistemology	Positivism	▪ Knowledge is hard, objective, tangible
▪ Humans initiate their own actions; as masters of free will	Voluntarism	human nature	Determinism	▪ Humans respond mechanically to environments, as marionettes
▪ The emphasis is on the particular and individual	Idiographic	methods	Nomothetic	▪ The emphasis is on discovery of general laws

Source: Burrell and Morgan (1979).

Note: Summaries at each end of box from Cohen and Manion (1994).

Table

Practice points

Write-ups should report the ‘mixing’, as appropriate, rather than reporting qualitative–quantitative components separately, and it remains helpful to acknowledge the pragmatism paradigm, given that this research is primarily ‘research question’-driven (modified by feasibility and context), rather than theory-driven per se. Write-ups should explicitly highlight what has been mixed, implicitly acknowledging the required rigour, expertise, time, and effort. Write-ups should explicitly discuss ‘mixing’ findings particularly.

Box 2. Literature search strategy: ‘What is mixed methods research, and how does it relate to medical education research?’

▪ Mainly used free-text in Web of Science (Science Citation Index Expanded, 1945-; Social Sciences Citation Index, 1956-; Arts and Humanities Citation Index, 1975-) to August 2010, with supplementary searches.

▪ Used: Medline, 1966-; AMED (Allied and Complementary Medicine), 1985-; British Nursing Index (BNI), 1985-; CINAHL (Cumulative Index to Nursing and Allied Health Literature), 1982-; International Bibliography of the Social Sciences and 1951-; ERIC (Educational Resources Information Center), 1966-; and PsychINFO, 1971-

▪ Combined the search terms: —mixed method* and ‘(medical) education* (research)’- - -; medical education and ∼∼∼qualitative and quantitative and mixed)∼∼∼ or ∼∼∼evidence-based∼∼∼ or ∼∼∼research paradigm∼∼∼ (*=‘wildcard’ character(s))

▪ Checked English Language titles (±abstracts) for articles reviewing mixed methods theory or practice (including relevance to being ‘evidence-based’) and examples in health care education (mostly undergraduate medical) or allied settings

▪ Searched: the University library catalogue for ‘mixed methods’ books; and used ancestry searching (from reference-lists), ad hoc ‘finds’, personal collection and January 2004–August 2010 handsearching of journals: Medical Education, Medical Teacher, Advances in Health Sciences Education, Academic Medicine, and Journal of Mixed Methods Research

Source: Haig and Dozier (2003a, 2003b).

Box 3. Four paradigms for considering knowledge claims and theory, the six key strands of the pragmatism continuum, and the ‘schism’ it spans

– ‘postpositivism: determination, reductionism, empirical observation and measurement, theory verification …

– constructivism: understanding, multiple participant meanings, social and historical construction, theory generation …

– advocacy/participatory: political, empowerment issue-orientated, collaborative, change-orientated …

– pragmatism: consequences of actions, problem-centred, pluralistic, real-world practice orientated’ (Creswell 2003, p. 6)

Six key strands of the ‘pragmatism’ continuum (Tashakkori & Teddlie 1998):

– Methods: use both qualitative and quantitative

– Logic: use both inductive and deductive

– Epistemology (knower↔known relationship): accept subjectivity and objectivity (‘epistemological relativism’)

– Axiology (the role of values): accept that values influence interpretation

– Ontology (nature of reality): accept ‘external reality’, but the best explanations for outcomes

– Causation: accept that causal relationships are possible, yet elusive

The schism that pragmatism spans:

‘ … realism versus idealism, foundational versus antifoundational, objective versus subjective, hard versus soft, scientists versus critics, personal versus impersonal, deductive reasoning versus inductive reasoning, rigor versus intuition, generalization versus uniqueness, logistic versus dialectic, rationalism versus naturalism, reductionism versus holistic, causal versus acausal, macro versus micro, correspondence versus coherence, quantifiers versus describers, and numbers versus words’. (Onwuegbuzie 2000, p. 13)

Box 4. The rationale for mixing quantitative and qualitative research

▪ Corroboration: of convergent findings, i.e. true ‘triangulation’ (Morse's (1991) concurrent validation of the same issue from different sources):

– Triangulation is a much misused term (Greene et al. 1989), often to claim rigour (Barbour 1998). Tashakkori and Teddlie (1998) noted Denzin's (1978) four basic types: data, investigator, theory, and methodological; and that ‘true triangulation’ was the ‘intellectual wedge that eventually broke the methodological hegemony of the monomethod purists’ (p. 41).

▪ Elaboration: enriching findings of one approach with the other (Greene et al's (1989) ‘complementarity’):

– Morse (1991) described simultaneous versus sequential methodological triangulation (depending on when mixing occurred), but contrasted simultaneous triangulation (of different aspects; complementary methods, ‘elaboration’) with true triangulation (concurrent validation: i.e. same concept, different methods; as confirmed by others (Kadushin et al. 2008)).

▪ Initiation: rethinking ideas, suggesting further work and seeking the provocative

▪ Development: one orientation informing the other (was a later addition)

Greene et al. (1989) used a fifth category, ‘expansion’ (of an evaluation project, e.g. exploring both process and outcome).

Reviewing 57 education evaluations, Greene et al. (1989) found that complementarity or expansion formed four-fifths of primary aims (and one-half of total aims), but combined data analysis was rare.

After Rossman and Wilson (1985).

Box 5. Contrasting contemporary views of medical education research

– ’It is very difficult to undertake meaningful research in education. The variables are too diffuse and difficult to identify. Very often they are not easy to measure. Other factors often contaminate the relationship between an educational event and its eventual outcome. Sometimes, particular outcomes are not easy to specify, nor are the timescales in which we might expect to see an effect or to see an effect last’. (Harden et al. 1999, p. 559)

– ‘ … High-stakes educational decisions regarding admission, promotion and accreditation may not be as life-threatening as life-and-death clinical decisions, but some of the consequences of these decisions certainly are irreversible and can affect individuals’ careers, quality (if not quantity) of life, job performance and productivity’. (Wolf 2000, p. 251)

– ‘Perhaps the highest level of professionalism in being a teacher is to contribute to the accumulation of evidence, to ‘problematize’ one's own educational situation, to start investigating and to engage in educational research’. (van der Vleuten et al. 2000, p. 249)

– [In response to Colliver's (2000) harsh critique of the educational evidence and theory underpinning problem-based learning (PBL)] ‘Does this mean … that all efforts at educational research are fruitless? Not at all. But we must take a cue from the natural sciences and move away from blind allegiance to the canons of sound methodology (randomization and all that) to recognition and support for research programmes, whose intent is to create an environment where ideas are shepherded from the basic science laboratory to the application setting’. (Norman & Schmidt 2000, p. 726)

– ‘Cognitive science is often said to be the basic science of medical education, and cognitive theory is routinely cited as a justification for educational practice. … a critical look at the theory [underpinning PBL] shows it is mostly metaphor, not rigorous, tested, confirmed scientific theory’. (Colliver 2002b, pp. 1217, 1220)

– ‘ … if the search for ‘replicable educational programmes’ is an implausible goal, if there are no meaningfully generalizable solutions in health professional education, then what is the purpose and value of science and scientific discourse in the health professional education field? [ … ] Education research is not rocket science, which is built on a structured, linear system with a straightforward set of factors which we can stick into a well-articulated formula to predict a clearly defined outcome. Rather, if we must make analogies to the physical sciences, we might do better to look to quantum mechanics and chaos theory. Such analogies will lead us away from the search for proofs of simple generalizable solutions to our collective problems, and towards the generation of rich understandings of the complex environments in which our collective problems are uniquely embedded’. (Regehr 2010, pp. 37–38)

Burrell G, Morgan G. Sociological paradigms and organisational analysis: Elements of the sociology of corporate life. Heinemann Educational Books, London 1979

 Google Scholar

Cohen L, Manion L. Research methods in education, 4th. Routledge, London 1994

 Google Scholar

Haig A, Dozier M. BEME Guide No 3: Systematic searching for evidence in medical education – Part 1: Sources of information. Med Teach 2003a; 25(4)352–363

 PubMed Web of Science ®Google Scholar

Creswell JW. Research design: Qualitative, quantitative, and mixed methods approaches, 2nd. Sage Publications, London 2003

 Google Scholar

Tashakkori A, Teddlie C. Mixed methodology: Combining qualitative and quantitative approaches. Sage, London 1998

 Google Scholar

Onwuegbuzie AJ, 2000. Positivists, post-positivists, post-structuralists, and post-modernists: Why can’t we all get along? Towards a framework for unifying research paradigms. Annual Meeting of the Association for the Advancement of Educational Research, Ponte Vedra, FL, 2000 November 18. 20p

 Google Scholar

Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nurs Res 1991; 40(2)120–123

 PubMed Web of Science ®Google Scholar

Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educ Eval Policy Anal 1989; 11(3)255–274

 Google Scholar

Barbour RS. Mixing qualitative methods: Quality assurance or qualitative quagmire?. Qual Health Res 1998; 8(3)352–361

 PubMedGoogle Scholar

Denzin NK. The research act: A theoretical introduction to sociological methods, 2nd. McGraw-Hill, New York 1978

 Google Scholar

Kadushin C, Hecht S, Sasson T, Saxe L. Triangulation and mixed methods designs: Practicing what we preach in the evaluation of an Israel experience educational program. Field Methods 2008; 20(1)46–65

 Google Scholar

Rossman GB, Wilson BL. Number and words: Combining quantitative and qualitative methods in a single large-scale evaluation study. Eval Rev 1985; 9(5)627–643

 Web of Science ®Google Scholar

Harden RM, Grant J, Buckley G, Hart IR. Best evidence medical education (BEME) Guide No. 1: Best evidence medical education. Med Teach 1999; 21(6)553–562

 PubMed Web of Science ®Google Scholar

Wolf FM. Lessons to be learned from evidence-based medicine: Practice and promise of evidence-based medicine and evidence-based education. Med Teach 2000; 22(3)251–259

 Web of Science ®Google Scholar

Colliver JA. Effectiveness of problem-based learning curricula: Research and theory. Acad Med 2000; 75(3)259–266

 PubMed Web of Science ®Google Scholar

Norman GR, Schmidt HG. Effectiveness of problem-based learning curricula: Theory, practice and paper darts. Med Educ 2000; 34(9)721–728

 PubMed Web of Science ®Google Scholar

Colliver JA. Educational theory and medical education practice: A cautionary note for medical school faculty [Review]. Acad Med 2002b; 77(12 Pt1)1217–1220

 PubMedGoogle Scholar

Regehr G. It's NOT rocket science: Rethinking our metaphors for research in health professions education. Med Educ 2010; 44(1)31–39

 PubMed Web of Science ®Google Scholar