Reliability of a seminar grading rubric in a grand rounds course

Figures & data

Table 1 Baseline student demographics

Demographic		Campus
		Amarillo (n = 55)	Dallas (n = 73)	Lubbock (n = 39)	P-valueamong campuses
Age (yrs)a	29.9 ± 5.8	31.2 ± 6.3	28.1 ± 4	28.3 ± 7	0.006b
Gender
Male (%)	44.6	49.1	35.6	53.8	0.124
Race
Caucasian (%)	60	72.7	42.5	76.9	<0.001c
Asian (%)	16	3.6	31.5	2.6
Hispanic (%)	17	20	13.7	17.9
Other (%)	7.2	3.6	12.3	2.6
Pre-enrollment	88.5 ± 4.1	87.4 ± 3.8	89.2 ± 4.3	88.6 ± 4	0.05d
GPA (%)a

Notes:

a Expressed as mean ± SD;

b Difference exists between Amarillo and Dallas;

c Dallas differs from Amarillo and Lubbock;

d Difference exists between Amarillo and Dallas.

Figure 1 Histogram depicting the differences in score between faculty graders.

Table 2 Grade difference of faculty pairs stratified by differences in academic rank

Difference in academic ranka	N	Mean grade differenceb	Std deviation	95% Confidence interval for mean
				Lower bound	Upper bound
0	83	4.36	3.03	3.6996	5.0233
1	78	4.69	4.07	3.7681	5.6047
2	59	4.95	4.07	3.8864	6.0051
3	32	3.89	2.91	2.8427	4.9385

Notes:

a Academic rank of faculty pair: 0 represents same rank, 1 represents a difference of one level of rank (eg, assistant professor and associate professor), 2 represents a difference of two levels of rank (eg, assistant professor and professor), 3 represents a difference of three levels of rank (instructor and professor);

b No difference in scores were noted across faculty ranks by one-way analysis of variance (P = 0.553).

Table 3 Correlation between student self-evaluation and faculty presentation scoresa (n = 252)

Grand rounds rubric component	Pearson correlation	P-value
Content scores	0.513	<0.001
Communication scores	0.455	<0.001
Overall presentation scores	0.539	<0.001

Note:

a Faculty presentation score was the student’s final presentation grade (ie, average of faculty grader 1 and 2 scores).

Figure 2 Differences between student self-evaluation and faculty presentation scores by quintile.^a

Notes: ^aFaculty presentation score was the student’s final presentation grade (ie, average of faculty grader 1 and 2 scores). Statistical significance assessed by Dunnett’s T3 post hoc test. A negative value indicates that the student’s score was less than the mean faculty grader score and a positive value indicates that it was greater; ^†Differs from quintiles 3 (P = 0.001), 4 (P < 0.001), and 5 (P < 0.001); ^‡Differs from quintiles 3 (P = 0.023), 4 (P < 0.001), and 5 (P < 0.001).

Table

Candidate:					Date:
Title of Presentation:					Start Time:
Evaluator:					End Time:
Directions: Use the scale below to assess each category. Enter the value in the “Points” box. **GRADER MAY AWARD PARTIAL VALUES**. Attach additional documentation for comments if needed.
Content Evaluation (60%)
Item	Outstanding (5 Points)	Meets Expectations (4 Points)	Needs Improvement (2.5 Points)	Unsatisfactory (0 Points)	Points	Factor	Yield
Topic Selection	Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate (not too broad or narrow) All 4 elements present	Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate (not too broad or narrow) 3 of 4 elements present	Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate (not too broad or narrow). 2 of 4 elements present	Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate 0 or 1 element present		5%	0
Objectives	All objectives clearly described and use measurable terms AND No overlap of objectives AND All objectives addressed AND Appropriate number of objectives (∼4)	Most objectives clearly described and use measureable terms Little overlap in objectives Most objectives addressed Number of objectives reasonable	Objectives unclear and ill defined Objectives overlap considerably in action verbs Most objectives not addressed Inappropriate # for presentation length	No objectives identified OR Objectives do not relate to presentation		5%	0
Introduction	Intro captured audience attention AND Thesis/purpose exceptionally clear AND Intro concise and well organized AND Provided clear overview of talk	Captured some of audience attention Thesis/purpose somewhat clear At times wordy or too brief; mostly organized Generally clear overview of talk	Did not capture audience attention Thesis/purpose not clear Too wordy or brief too and vague Preview of talk confusing and disorganized	No introduction presented in talk OR Intro not relevant to presentation		10%	0
Organization	Concise and complete intro and conclusion AND Clear and logical progression throughout AND All facts linked to topic and objectives AND All major points highlighted	Somewhat brief introduction and conclusion Mostly clear and logical progression Most facts linked to topic and objectives Most major points highlighted	Minimal intro and conclusion Progression throughout difficult to follow Little link between facts and topic/objectives Major points sparsely highlighted	No introduction or conclusion used No logical progression of ideas Facts not linked to topic and objectives Major points not highlighted		20%	0
Primary Literature Citation and Analysis	Comprehensive incorporation of primary literature with most relevant/timely references elaborated upon AND Analysis of literature and/or trial design insightful and accurate	Most key primary literature cited and incorporated Most literature current/timely Analysis of literature and/or trial design limited to provided author’s conclusion(s)	Little primary literature used in talk Some key articles missing Much literature out-of-date Little analysis of literature and/or trial design; recited data	Relied on secondary or tertiary literature (key primary literature missed) OR No current literature cited OR No analysis of literature and/or trial		20%	0
Statistical Interpretation of Data	Tests named, explained, justified, and critiqued with alternative tests identified if appropriate AND Number needed to treat (NNT) or harm (NNH) calculated for all appropriate data	Tests named, explained, and justified if appropriate NNT or NNH calculated for some data if appropriate	Tests named but not explained or justified if appropriate No NNT or NNH calculated if appropriate	No statistical tests named if appropriate		10%	0
Analysis and Application in Practice	Addressed both contemporary and future practice AND Gave well thought-out, detailed recommendation on how to apply including additional data needed Broad perspective given	Addressed both contemporary and future practice Perspective limited	Addressed only one specific setting or perspective Superficially addressed setting and/or perspective	Did not address a specific setting or perspective		15%	0
Response to Questions	All questions were answered correctly AND Was able to justify answers AND Paraphrased understanding of all questions	Majority of questions were answered correctly Most answers were justified Paraphrased understanding of most questions	Majority of questions were only partially answered or not answered correctly Majority of answers poorly justified Paraphrased few questions	Questions were not answered or justified Paraphrased understanding of no questions		15%	0
				Content Grade			0%
Communication Skills Evaluation (40%)
Professionalism	Exceptionally dressed (business attire) AND Formal tone and attitude displayed AND May serve as a positive role model for future presenters	Mostly appropriately dressed Acceptable tone and attitude displayed Possibly could serve as a positive role model for future presenters	Somewhat appropriately dressed Tone and attitude too informal Questionable ability to serve as a positive role model for future presenters	Inappropriately dressed Would not serve as a positive role model for future presenters		10%	0
Transitions	All transitions between major areas in talk exceptionally clear and appropriate AND Brief summaries of key points provided for all major topic areas	Most transitions between major areas in talk clear and appropriate Brief summaries of key points provided for most major topic areas	Few transitions between major areas in talk clear and appropriate Few brief summaries of key points provided for major topic areas	Transitions between major areas in talk unclear inappropriate No brief summaries of key points provided for major topic areas		15%	0
Slides and Graphics	Amount of material on slide facilitated understanding of presentation AND Slides contained quality pictures, diagrams, tables, and/or animations AND Slide background and font were professional and enhanced readability AND Slides free from typos and grammatical errors, abbreviations defined	Some slides contained too much or too little information Slides mostly text, some inclusion of a few basic tables, diagrams, or clip art as pictures Slide background and font was acceptable and readable Mostly free from typos and grammatical errors, most abbreviations defined	Most slides contained too much or too little information Slides consisted almost entirely of text; tables, diagrams, or pictures rarely used Background and font unprofessional and/or distracting and/or compromised readability Many typos and grammatical errors, few abbreviations defined	All slides contained either too much or too little information OR All slides were text; no tables, diagrams, or pictures used OR Slide background and font was unreadable and completely distracting OR Many errors and unreadable		20%	0
Presentation Style	Maintains eye contact with audience AND Rarely returns to notes AND Exceptional and consistent facial expressions, gestures, and posture. No distracting movements or gestures	Eye contact maintained most of the time Returns to notes occasionally Acceptable facial expressions, gestures, and posture. Minimal distracting movements or gestures	Eye contact made rarely Most of presentation read Inconsistent and incongruent facial expressions, gestures, and posture Some distracting movements or gestures	Does not make eye contact Reads entire presentation Consistently poor and incongruent facial expressions, repetitive, distracting gestures, and poor posture		20%	0
Elocution	Always articulate with no pronunciation or grammatical errors AND Always uses correct medical/scientific nomenclature AND All word fillers (eg, “um”) appropriate and not distracting AND All attendees can hear presentation Rate of speech ideal	Mostly articulate with few (2–3) pronunciation or grammatical errors Rarely uses incorrect medical/scientific nomenclature Word fillers mostly appropriate and rarely distracting Most attendees can hear presentation Rate of speech slightly too fast or slow	Mostly inaudible and inarticulate with several (3–5) pronunciation or grammatical errors Frequently uses incorrect medical/scientific nomenclature Word fillers frequent and distracting Many attendees can not hear presentation Rate of speech significantly too fast or slow	Inaudible and nonarticulate with numerous errors (>5) OR Constantly uses incorrect medical/scientific nomenclature OR Constant use of word fillers Rate of speech so fast or slow that presentation is not comprehendible		15%	0
Accuracy and Completeness of References	Bibliography complete, in proper format, and no errors AND All graphs, charts, and tables appropriately referenced	Bibliography mostly complete, in proper format, with few (<2) errors Most graphs, charts, and tables appropriately referenced	Bibliography mostly incomplete, not in proper format, with several (>2) errors Most graphs, charts, and tables not appropriately referenced	No bibliography provided OR No graphs, charts, and tables were appropriately referenced		5%	0
Time Management (Goal 40 min, Excludes Q and A)	Spends an appropriate amount of time on the major sections of the presentation AND Presentation within 2.5 minutes of target	Spends an appropriate amount of time on a majority of the major sections of the presentation Presentation within 2.5–5 minutes of target	Spends an inappropriate amount of time on the majority of sections of the presentation (too much or too little) Presentation within 5–10 minutes of target	Inappropriate time spent on all of major sections. Presentation <10 min of target results in 69% grade. If >10 min presentation to be stopped and graded as-is		15%	0
				Communication Grade			0%
Strengths of Presentation:
_____________________________________________________________________________________________________________________________ _____________________________________________________________________________________________________________________________
Weaknesses of Presentation:
_____________________________________________________________________________________________________________________________ _____________________________________________________________________________________________________________________________
				Overall Presentation Grade: 0%