Supervisors’ untrained postgraduate rubric use for formative and summative purposes

Figures & data

Figure 1. Distribution of internship rubric use.

a. 59 internships with solely formative rubric use (59 rubric forms); b. 76 internships with both formative and summative rubric use (resulting in 152 rubric forms); c. 102 internships with solely summative rubric use (59 rubric forms)

Table 1. Descriptive statistics for criterion scores used and omitted in the rubric assessment of research skills for formative and summative use

Rubric criterion	Formative rubric use						Summative rubric use
	Completed criteria				Omitted		Completed criteria				Omitted
	n	Range	Mean	SD	n	(%)	n	Range	Mean	SD	n	(%)
Performing Research (PR)
Design research	127	1.0 - 3.0	2.11	.50	8	5.9	175	1.0 - 3.0	2.27	.54	3	1.7
Data analysis and  interpretation	113	1.0 - 3.0	2.28	.51	22	16.3	178	1.0 - 3.0	2.35	.50	0	0.0
Discussing outcomes	129	1.0 - 3.0	2.23	.42	6	4.4	177	1.0 - 3.0	2.33	.50	1	0.6
Practical Skills (PS)
Technical skills	132	1.0 - 3.0	2.32	.43	3	2.2	178	1.0 - 3.0	2.51	.44	0	0.0
Efficiency	121	1.0 - 3.0	2.36	.56	14	10.4	171	1.0 - 3.0	2.51	.55	7	3.9
Organization work records	128	1.0 - 3.0	2.20	.39	7	5.2	172	1.0 - 3.0	2.36	.42	6	3.4
Organization working place	124	1.0 - 3.0	2.38	.41	11	8.1	171	1.0 - 3.0	2.48	.44	7	3.9
Professional attitude (PA)
Initiative, independence, creativity & handling feedback	135	1.5 - 3.0	2.36	.37	0	0.0	178	1.0 - 3.0	2.44	.42	0	0.0
Critical attitude	127	1.0 - 3.0	2.19	.43	8	5.9	174	1.0 - 3.0	2.27	.50	4	2.2
Integrity & conscientiousness	124	2.0 - 3.0	2.04	.20	11	8.1	164	1.5 - 3.0	2.05	.22	14	7.9
Perseverance & dedication	119	1.0 - 3.0	2.32	.48	16	11.9	169	2.0 - 3.0	2.57	.48	9	5.1
Communication with colleagues	129	2.0 - 3.0	2.55	.45	6	4.4	174	1.5 - 3.0	2.63	.45	4	2.2
Timelyness	131	1.0 - 3.0	2.46	.48	4	3.0	177	1.0 - 3.0	2.54	.49	1	0.6

n: number of cases, SD: standard deviation. Total n formative use = 135, Total n summative use = 178. Total of completely filled out rubrics for formative use = 87, Total of completely filled out rubrics for summative use = 140.

Table 2. Number of missing criteria within the incomplete rubrics.

Number of missing criteria	Formative rubric use		Summative rubric use
	n	%	n	%
1	19	39.6	27	71.0
2	12	25.0	5	13.2
3	5	10.4	5	13.2
4	5	10.4	1	2.6
5	4	8.3
6	3	6.3
total	48^a	100	38^b	100

a: 48 of a total of 135 rubrics for formative use were incomplete (35.6%), b: 38 of a total of 178 rubrics for summative use were incomplete (21.3%)

Table 3. Multiple linear regression model.

	B	SE B	β	p	95% CI
(Constant)	3.632	.254		.000	[3.130, 4.134]
Performing Research (PR)
Design research*	.285	.095	.194	.003*	[.10, .47]
Data analysis and interpretation	.100	.106	.060	.348	[-.11, .31]
Discussing outcomes	.141	.102	.087	.169	[-.06, .34]
Practical Skills (PS)
Technical skills	.051	.112	.028	.651	[-.17, .27]
Efficiency*	.206	.088	.139	.021*	[.03, .38]
Organization work records*	-.247	.110	-.130	.026*	[-.47, −.03]
Organization working place	.106	.104	.058	.309	[-.10,.31]
Professional attitude (PA)
Initiative, independence, creativity, handling feedback*	.413	.124	.211	.001*	[.17, .66]
Critical attitude	.101	.097	.063	.298	[-.09, .31]
Perseverance & Dedication*	.225	.086	.132	.010*	[.06, .40]
Communication with colleagues	.179	.103	.100	.084	[-.02, .38]
Timelyness*	.309	.103	.191	.003*	[.11, .51]
R^2	.77
ΔR^2	.75

PR: Performing research; PS: Practical Skills; PA: Professional attitude.

B regression beta, p significance value, SE B standard error unstandardised beta, β standardised beta, 95% CI 95% confidence interval, R² variance explained, ΔR² adjusted variance explained.

* p < .05

Table

Criteria	Insufficient	Satisfactory	Excellent
Performing Research (PR)
Design research plan /experiments	Executes plans devised by supervisor only	Proposes new valid experiments based on previous results	Proposes many new, relevant experiments (with proper controls)
		Has creative ideas	“Owns” the project, has original, creative ideas
Data analysis and interpretation	Depends on supervisor for correct interpretation of results	Provides correct analysis interpretation of results at later stages of the project	Provides correct analysis and interpretation of results from the start of the project
	Invalid statistical analysis	Statistical analysis correct	Recognizes implications
Discussing research outcomes	Hardly participates in discussions	Participates in discussions	Is critical and occasionally leading during discussions
	Fails to place research into perspective	Discussion in the light of (recent) literature	Stays on top of recent literature
Practical skills (PS)
Technical skills	Fails to master technical/lab skills	Masters required technical/lab skills	Has excellent technical skills
	Fails to apply techniques independently	Applies techniques independently	Finds and masters new technical approach, improves existing procedures
Efficiency	Waiting times in protocols are spent inefficiently	Uses waiting times for preparing buffers, reading etc.	Runs parallel experiments to use time efficiently and effectively
Organization lab journal /log/work records	Badly organized	Well organized
	Required information is missing	All required information is available	Repetition of experiments based on  information provided easily possible
Organization working place	Workplace is a mess	Tidies workplace regularly	Workplace is always clean
	Fails to clean equipment after use	Cleans equipment after use	Equipment is always clean
	Does not follow guidelines and protocols	Follows guidelines and protocols	Suggests improvements for protocols
Professional attitude (PA)
Initiative, independence, creativity, handling feedback	Many feedback sessions are required	Regular feedback sessions were needed	The amount of feedback needed was minimal
	Relies on supervisor’s instructions only	Takes initiative (initially) after stimulation	Consults experts outside the group in consultation with supervisor, designs large parts of the project
			Finds relevant new literature
	Minimal improvement based on feedback	Feedback led to reasonable improvement	Response to feedback yielded excellent improvements
Critical attitude	Critical attitude is absent Self-reflection is absent	Shows self-reflection and has critical attitude towards (published) research	Critical attitude is based on intellectual depth and profundity
Integrity,  Conscientiousness	Data manipulated or left out**	Accurate, reliable and trustworthy, shows awareness of confidentiality of information
Perseverance, Dedication	Loses motivation when experiments / research fail(s)	Repeats experiment until satisfactory result is obtained	Perseveres, but knows when to stop
Communication with colleagues	Thinks he/she is the only worker in the lab	• Takes (needs of) colleagues into account • Communicates with colleagues, e.g. to share equipment	• Knows when to ask questions • Accepts, communicates and learns from own failures
Timelyness	Fails to meet deadlines	Meets most deadlines	Sets own deadlines and adheres to them
	Fails to keep appointments	Keeps appointments	Schedules appointments when necessary