The Quality Of Evidence In Preclinical Medical Education Literature: A Systematic Review

Figures & data

Table 1 Scoring Scheme Used For Analysis Of Articles Meeting Inclusion Criteria (Modified From The Medical Education Research Study Quality Index¹³)

Domain	Attribute	Score
Number of cohorts	Not applicable/could not assess	0
	1	1
	>1	2
Number of learners	Not applicable/could not assess	0
	<10	1
	10–19	2
	20–49	3
	50–100	4
	>100	5
Study design	Not applicable/could not assess	0
	Single group cross sectional or single group posttest only	1
	Single group pretest and posttest	2
	Nonrandomized, 2 groups	3
	Randomized controlled trial	4
Number of institutions	Not applicable/could not assess	0
	1	1
	2	2
	>2	3
Response rate	Not applicable	0
	<50% or not reported	1
	50–74%	2
	At least 75%	3
Type of data	Not applicable/could not assess	0
	Assessment by study participant	1
	Objective measurement	2
Appropriateness of data analysis	Not applicable/could not assess	0
	Inappropriate for study design or type of data	1
	Appropriate for study design or type of data	2
Complexity of analysis	Not applicable/could not assess	0
	Descriptive only	1
	Beyond descriptive analysis	2
Outcomes	Not applicable/could not assess	0
	Satisfaction, attitudes, perceptions, opinions, general facts	1
	Knowledge or skills	2
	Behaviors	3
	Patient/health care system	4

Note: Reproduced with permission from Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298(9):1002–1009. Copyright 2007, American Medical Association. All rights reserved.¹³

Figure 1 Literature search results and proportion of studies included in the analysis.

Figure 2 Types of intervention described in analyzed articles.

Note: Multiple interventions were used in 57 manuscripts.

Table 2 Overall Scoring Data For Analyzed Manuscripts

	Anatomy	Physiology	Biochemistry	Total
N	75	86	16	177
Mean total score	16.6	14.5	16.1	15.7
Standard deviation	3.9	4.4	1.65	3.3
Minimum	3	2	13
Maximum	23	21	20

Table 3 Percentages Of Manuscripts With A Given Score In Each Domain Of Assessment

Criterion	Score	Anatomy	Physiology	Biochemistry	P
Number of cohorts
Could not assess	0	32.0%	14.0%	12.5%
1	1	64.0%	73.3%	75.0%
>1	2	4.0%	12.8%	12.5%
Mean score		1.28	0.98	1	<0.01
Number of learners
Could not assess	0	1.3%	8.2%	0%
<10	1	0.0%	0.0%	0%
10 to 19	2	5.3%	2.3%	0%
20 to 49	3	14.8%	9.5%	6.3%
50 to 99	4	13.3%	23.5%	25.0%
At least 100	5	65.3%	56.5%	68.8%
Mean score		4.35	4.09	4.63	0.20
Study design
Could not assess	0	2.7%	9.4%	0.0%
Single group cross-sectional or single group posttest only	1	30.7%	43.5%	50.0%
Single group pre/posttest	2	8.0%	10.6%	18.8%
Non-randomized, 2 group	3	49.3%	30.6%	25.0%
Randomized controlled trial	4	9.3%	5.9%	6.3%
Mean score		2.31	1.79	1.79	0.01
Number of institutions
Could not assess	0	0.0%	1.2%	0%
1	1	100.0%	97.6%	100%
2	2	0.0%	1.2%	0%
>2	3	0.0%	0.0%	0%
Mean score		1	0.99	1	0.84
Response rate
Not applicable	0	22.7%	19.8%	37.5%
<50% or not reported	1	9.3%	32.6%	18.8%
50–74%	2	10.7%	10.4%	0%
At least 75%	3	57.3%	37.2%	43.7%
Mean score		2.03	1.65	1.5	0.10
Type of data*
Could not assess	0	10.6%	5.9%	0%
Assessment by participant	1	42.7%	43.5%	68.8%
Objective measurement	2	46.7%	50.6%	31.3%
Mean score		1.36	1.43	1.31	0.69
Appropriateness of data analysis
Could not assess	0	6.6%	30.2%	0%
Inappropriate to study	1	50.7%	41.9%	31.3%
Appropriate for study	2	42.7%	27.9%	68.8%
Mean score		1.36	0.98	1.69	<0.01
Complexity of analysis
Could not assess	0	8.0%	31.4%	0%
Descriptive analysis only	1	37.3%	24.4%	37.5%
Beyond descriptive analysis	2	54.7%	44.2%	62.5%
Mean score		1.46	1.13	1.63	0.01
Outcomes
Could not assess	0	6.7%	5.8%	0%
Satisfaction, attitudes, perceptions, and opinions	1	48.0%	43.0%	56.2%
Knowledge, skills	2	41.3%	51.2%	43.8%
Behaviors	3	4.0%	0.0%	0%
Patient/health care outcomes	4	0.0%	0.0%	0%
Mean score		1.42	1.45	1.44	0.96

Note: *Some studies included multiple measures of effectiveness, such as a satisfaction survey combined with a pretest/posttest.

Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298(9):1002–1009. doi:10.1001/jama.298.9.1002

 PubMed Web of Science ®Google Scholar