Examiners’ use of rubric criteria for grading bachelor theses

Figures & data

Figure 1. Calculated Euclidean distance hierarchically clustering matrix, indicating correlations between the scores of all subcriteria of all 13 categories of all numerated rubrics. The color range in the legend indicates the Pearson correlations between the subcategories (pink = Pearson correlation of 0.0, green = Pearson correlation of 1.0). The subcriteria are ordered based on the similarity in scores, also indicated by the dendogram on the left. Hence, subcriteria that are scored similarly, such as the discussion overview and discussion line of thought, are directly connected in this dendogram. The dendogram is calculated based on Euclidean distance hierarchically clustering.

Figure 2. Trendline (blue) of the correlation between thesis grade and the mean score of the 13 subcriteria of the assessed rubric. Boxes behind the trendline indicate the median and variation in rubric score per grade. Shown are the boundaries of the second and third quartile of the data distribution. Black bars within the boxes indicate the median and whiskers the Q1 and Q4 values within 1.5 times the interquartile range. Individual dots are outliers beyond the Q1 and Q4 intervals.

Figure 3. Multiple linear regression analysis of thesis grade and criteria scores for all theses and for theses graded as sufficient (≤7.0), good (<7.0 and ≤8.0), and excellent (>8.0). The grey bars indicate standard deviations.

Table 1. Descriptive statistics for criteria scores of all rubrics used in the assessment of bachelor theses in biology.

Rubric criterium	N^a	Range	Mean	S.D.	r	R^2adj^b
Title	296	1.0–3.0	2.27	0.48	0.48	0.23**
Abstract	294	1.0–3.0	2.29	0.56	0.59	0.34**
Introduction	296	1.0–3.0	2.43	0.52	0.61	0.37**
Scope	281	1.0–3.0	2.32	0.60	0.56	0.32**
Structure	296	1.0–3.0	2.39	0.50	0.76	0.58**
Discussion	291	1.0–3.0	2.31	0.49	0.72	0.52**
Scientific quality	295	1.0–3.0	2.44	0.44	0.79	0.62**
Spelling and grammar	290	1.0–3.0	2.49	0.53	0.50	0.24**
Style	296	1.0–3.0	2.39	0.46	0.72	0.52**
Length and layout	293	1.0–3.0	2.42	0.51	0.59	0.34**
Figures	296	1.0–3.0	2.36	0.53	0.62	0.38**
References	296	1.5–3.0	2.58	0.43	0.61	0.37**
Attitude	287	1.0–3.0	2.52	0.46	0.73	0.53**

^a The number of category scores differs per rubric category because of missing data.

^b *p < 0.01;

** p < 0.001.

Table 2. Regression analysis of thesis grade bins and criteria scores adjusted for interdependence between criteria scores. The regression analysis is performed separately for three bins; sufficient thesis grades (left column, <7.0; n = 48), good thesis grades (grades ≥ 7.0 to <8.5; n = 158) and excellent thesis grades (grades ≥ 8.5; n = 55).

		Sufficient thesis grade bin (≤7.0)					Good thesis grade bin (>7.0 to ≤8.0)					Excellent thesis grade bin (>8.0)
	Rubric criterion	N^a	S.D.	Mean	r	R^2adj^b	N^a	S.D.	Mean	r	R^2adj^b	N^a	S.D.	Mean	r	R^2adj^b
Opening	Title	93	0.29	2.00	0.10	0.00	144	0.47	2.28	0.21	0.04	59	0.48	2.64	0.17	0.01
	Abstract	93	0.38	1.89	0.14	0.01	142	0.52	2.36	0.29	0.08**	59	0.47	2.75	0.37	0.13*
	Introduction	93	0.43	2.03	0.25	0.05	144	0.46	2.53	0.39	0.15**	59	0.32	2.82	0.24	0.04
	Scope	85	0.54	1.89	0.33	0.10*	139	0.52	2.41	0.11	0.00	57	0.45	2.78	0.29	0.07*
Content	Structure	93	0.37	1.89	0.48	0.22**	144	0.38	2.53	0.48	0.23**	59	0.23	2.84	0.24	0.04
	Discussion	92	0.40	1.92	0.40	0.15**	141	0.39	2.36	0.54	0.29*	58	0.32	2.81	0.32	0.09
	Scientific quality	92	0.34	2.00	0.55	0.30**	144	0.32	2.55	0.50	0.25**	59	0.22	2.87	0.38	0.13*
Presentation	Spelling and grammar	91	0.50	2.16	0.24	0.05	140	0.49	2.58	0.27	0.07*	59	0.41	2.80	0.27	0.06
	Style	93	0.35	1.97	0.42	0.17**	144	0.36	2.48	0.39	0.15*	59	0.30	2.81	0.38	0.13*
	Length and layout	91	0.44	2.03	0.22	0.04	144	0.45	2.51	0.35	0.12**	58	0.34	2.79	0.20	0.02
	Figures	93	0.42	1.99	0.33	0.10*	144	0.50	2.41	0.38	0.14**	59	0.30	2.82	0.27	0.06
	References	93	0.37	2.22	0.27	0.06*	144	0.37	2.69	0.28	0.07**	59	0.22	2.88	0.16	0.00
Attitude	Professional attitude	89	0.39	2.08	0.53	0.27**	141	0.34	2.64	0.39	0.15*	57	0.18	2.91	0.10	0.00

Note. N is total number of cases, S.D. is standard deviation, R²_adj is explained variance of thesis grade adjusted for the number of predictors

^a The number of criteria scores differs per rubric category because of missing data.

^b *p < .01;

** p < .001.

Figure 4. Frequency distribution of criteria scores of the 13 rubric categories for theses graded as sufficient (≤7.0), good (<7.0 and ≤8.0), and excellent (>8.0). Criteria scores are color coded from red (criteria scores from 1 to 1.22) to green (criteria scores from 2.78 to 3).

Figure 5. Average rubric criteria scores (rows) of all 318 assessed biology bachelor theses (columns). The 318 bachelor theses are sorted in columns from lowest thesis grade (left) to highest thesis grade (right) with thick black vertical lines marking the boundaries of the three separate grade bins: sufficient (≤7.0), good (<7.0 and ≤8.0), and excellent (>8.0). The criteria scores of each thesis are presented with color codes (red = sufficient, yellow = good and green = excellent). Criteria scores that were missing are marked in white. The criteria are ordered based on the similarity in scores, as indicated by the dendogram on the right. Criteria that are scored similarly, such as quality and structure or references and attitude, are directly connected in this dendogram. The dendogram is calculated based on Euclidean distance hierarchically clustering.

Appendix 1. Rubric used for assessing Biology bachelor theses at Utrecht University.

Criteria	Insufficient	Sufficient	Good
Title	Title does not justify the content Title is too long	Title justifies the content	Title justifies the content and is catching
Abstract	Abstract is missing Abstract is unclear Abstract lacks, or gives too much, irrelevant information	The abstract is present In general the abstract is clear The abstract contains all important elements	Abstract is present Abstract contains a clear, concise and balanced description of all important elements.
Introduction and scope of thesis	The introduction contains incomplete information The introduction contains a lot of irrelevant information The scope of the thesis is missing The scope of the thesis is unclear	The introduction contains sufficient information The introduction is functional Most of the presented information is relevant Some repetition is present, but this is not disturbing The scope of the thesis is clear	The introduction contains sufficient relevant, information to put the scope in a broader perspective The introduction is catchy and inviting to continue reading The scope of the thesis is well defined
Structure	The order of paragraphs is illogical. Transitions between the paragraphs are lacking The different sections are not in balance Not all of the information is consistent with the scope Information is frequently repeated The line of thought is difficult to follow	The structure and order of most paragraphs is sound Transitions between paragraphs are mostly logical Most sections are well balanced Most of the information is consistent with the scope Information is sometimes unnecessarily repeated The line of thought is clear	The structure and order of the paragraphs is sound Smooth transitions between paragraphs The sections are well balanced The information is consistent with the scope Only if necessary, the information is repeated The line of thought is clear and well structured
Discussion/ Conclusion	The discussion is missing or too short The discussion lacks depth The discussion does not go back to the scope of the thesis No conclusions are drawn The conclusions are not supported by the presented findings	The discussion ties together loose ends The discussion has limited depth The discussion goes back to the scope of the thesis Conclusions are drawn in line with the presented findings	Discussion demonstrates a good overview of the subject of the thesis In depth discussion The discussion goes back to the scope of the thesis Conclusions are drawn in line with presented findings
Scientific quality	The thesis does not demonstrate sufficient knowledge of the field Results from the referred articles are not discussed correctly Arguments are not supported by evidence It is difficult to distinguish facts from hypotheses Information is not well integrated: the text often reads as loose pieces of information	The thesis shows sufficient knowledge of the field in most parts Most important results from the references are discussed correctly Some arguments are supported by evidence Facts and hypotheses are not always easy to distinguish from each other Limited integration of information	The thesis shows a high level of knowledge and understanding of the field Results from the references are adequately discussed Arguments are well explained and scientifically supported Facts and hypotheses can easily be distinguished Results from different papers are integrated well into new insights
Spelling and grammar	Disturbing spelling or grammar mistakes, which complicates understanding of the text	There are errors in spelling and/or grammar, but this does not prevent understanding the text.	None, or very few errors in grammar or spelling
Style	Writing style is hardly scientific Many sentences are not structured properly, or too long or too short Little variation in vocabulary Important terms are not explained Bad use of punctuation	In general the writing style is scientific Few sentences are badly structured, but this is not too disturbing In general a sufficient vocabulary is used Important terms are explained Use of punctuation is usually correct and makes the text easier to follow	Style is concise, and scientific. It is a pleasure to read the text Sentences flow smoothly and are not too long or too short Rich vocabulary Important terms are well explained Good use of punctuation
Length/Layout	The length is too short / too long The layout is sloppy and not appealing	The length is as required The layout is uniform and organized	The length is as required The layout is uniform and well organized
Figures	Insufficient number of figures Figures are often of bad quality Too many irrelevant figures Figure legends are missing or badly formulated The figures are not referred to in the text or the reference is wrong A clear explanation of the figure in the text is missing	Sufficient number of figures Most figures are of good quality Most figures are relevant Figure legends are adequate to understand the figure Good reference to the figures in the text Most figures are explained well in the text	Sufficient number of figures, that support the text Figures are of good quality Figures are relevant and when needed adjusted or specifically designed for the thesis Figure legends are complete Good reference to the figures in the text Figures are explained well in the text
References	Insufficient number of references (<20) An insufficient number of primary literature is used Finding and selecting references was mainly done by the supervisor None, or incorrect referral to references in the text The reference list is incomplete or sloppy	Sufficient number of references (>20) A sufficient amount of primary literature was used Most sources were found by the student Most of the articles are correctly referred to in the text The reference list is largely complete, clear and uniform, with some minor mistakes	Sufficient number of references (>20) A sufficient amount of primary literature was used All references were found by the student The articles are correctly referred in the text The reference list is complete, clear and uniform
Professional attitude	A lot of supervision was required Minimal improvement after feedback The student did not put enough effort into the thesis work The student failed to follow the agreements The student failed to meet the deadlines	Regular feedback sessions were required Feedback led to reasonable improvements The student was committed The student fulfilled the agreements The students met the agreed deadline	Minimal feedback was required, the thesis is a true product of the student Response to feedback yielded excellent improvements The student was enthusiastic and committed The student fulfilled the agreements The student met the agreed deadline