Knowledge, application and how about competence? Qualitative assessment of multiple-choice questions for dental students

Figures & data

Table 1. Moore’s expanded outcomes framework (2009)

Outcomes Framework	Millers’ Framework	Description
Participation Level 1		The number of physicians and others who participated in the educational activity
Satisfaction Level 2		The degree to which the expectations of the participants about the setting and delivery of the educational activity were met
Learning: Declarative Knowledge Level 3a	Knows	The degree to which participants state what the educational activity intended them to know
Learning: Procedural Knowledge Level 3b	Knows How	The degree to which participants state how to do what the educational activity intended them to know how to do
Learning: Competence Level 4	Shows How	The degree to which participants show in an educational setting how to do what the educational activity intended them to be able to do
Performance Level 5	Does	The degree to which participants do what the educational activity intended them to be able to do in their practices
Patient Health Level 6		The degree to which the health status of patients improves due to changes in the practice behaviour of participants
Community Health Level 7		The degree to which the health status of a community of patients changes due to changes in the practice behaviour of participants

Table 2. Frameworks – From Miller to Bloom to Moore

	Definition of LEVELS
	Level 1/Level 3a	Level 2/Level 3b	Level 3/Level 4
Moore’s Expanded Outcomes Framework	Declarative Knowledge: The acquisition and interpretation of facts	Procedural Knowledge: Participants who possess procedural knowledge can describe how to do something but may not be able to actually do it	Competence: Participants show in an educational setting how to do what the educational activity intended them to be able to do
Millers’ Pyramid	Knows	Knows How	Shows How
Blooms’ Taxonomy	Knowledge of Learning	Comprehension	Problem-Solving
Anderson & Krathwohl 2001 (revised version of Blooms’ Taxonomy)	Knowledge & Remembering: Learner must recall or recognize specific information	Analysis & Application: Learner must understand or be able to explain in own words previously learned information and use new information, rules, methods, concepts, principles, and theories	Synthesis & Evaluation: Transferring existing knowledge and skills to new situations, understanding a concept’s components and their relationships to each other, and analyse information
Cognitive Levels of National Board of Dental Examination	Understanding: Items elicit knowledge of specific facts, terminology, sequences, methodology, principles, theories, and structures in a different context.	Application: Items elicit application of specific facts, terminology, sequences, methodology, principles, theories, and structures in a complex manner.	Reasoning: Items elicit understanding or the ability to identify and interpret specific data, terminology, sequences, methodology, principles, theories, and structures.

Figure 1. Example of coding from (insert after ‘Evaluation of MCQs’ in methods)

Table 3. Comparison of cognitive levels of UCD questions with cognitive levels of NBDE questions

Cognitive Levels	Number of UCD Questions	Number of NBDE Questions	Total Number of MCQs
Comparison of UCD questions (2002–2009) with NBDE questions (2000–2008)
Level 3a	47	73	120
Level 3b	3	26	29
Level 4	0	1	1
	50	100	150

*χ2 = 9.234

**P-value = 0.005

There was a significant difference between the cognitive levels of UCD and NBDE questions.

Table 4. Comparison of cognitive levels of MCQs (2002–2009) with MCQs (2010–2018)

Cognitive Levels	Number of MCQs 2002-2009	Number of MCQs 2010-2018	Total Number of MCQs
Development of UCD questions overtime
Level 3a	47	32	79
Level 3b	3	17	20
Level 4	0	1	1
	50	50	100

*χ 2 (1, N = 150) = 0.004972

**P-value < .05. There was a significant difference between the cognitive levels of UCD and NBDE questions.

Table 5. The role of various subject areas on cognitive levels of MCQs

	Subject Areas
	Cariology; Dental restoration; Endodontics n – (%)	Periodontology; Prophylaxis n – (%)	Restorative dentistry; Permanent dentures n – (%)	Prosthetic fundaments; Removable prosthetics n – (%)	Surgery n – (%)	Orthodontic n – (%)	Total MCQs n
Distribution of the cognitive levels of UCD questions by subject areas
Level 3a	13 – (72)	6 – (86)	22 – (88)	10 – (59)	16 – (94)	12 – (75)	79
Level 3b	4 – (22)	1 – (14)	3 – (12)	7 – (41)	1 – (6)	4 – (25)	20
Level 4	1 – (6)	0	0	0	0	0	1
							100

* χ2 = 13.021

**P-value = 0.188

There was no significant associate between cognitive levels and subject area.

Table 6. Association between cognitive levels of MCQs with items’ difficulty level

	Difficulty Level of MCQs
	0.1–0.2	0.2–0.3	0.3–0.4	0.4–0.5	0.5–0.6	0.6–0.7	0.7–0.8	0.8–0.9	0.9–1
	n	n	n	n	n	n	n	n	n	Total
Distribution of the cognitive levels of UCD questions by difficulty level
Level 3a	2	3	5	7	17	16	16	10	3	79
Level 3b	0	2	2	2	5	1	2	5	1	20
Level 4	0	0	0	0	1	0	0	0	0	1
										100

*χ2 = 7.278

**P-value = 0.534

There was no significant associate between cognitive levels and difficulty level of questions.