Reviewer agreement in scoring 419 abstracts for scientific orthopedics meetings

Figures & data

Table 1.  Interclass correlation coefficients and mean ISK abstract scores (N = 419)

Year: month	na	nr	ICC (95% CI)	ISK score mean (95% CI)
2001: V	62	8	0.94 (0.92–0.96)	55 (51–59)
2001: X	11	8	0.90 (0.77–0.97)	53 (47–59)
2002: V	10	6	0.83 (0.60–0.95)	58 (50–66)
2002: X	14	7	0.91 (0.79–0.97)	59 (52–66)
2003: I	45	7	0.87 (0.79–0.93)	60 (57–63)
2003: V	15	6	0.82 (0.64–0.93)	55 (49–60)
2003: X	14	6	0.83 (0.63–0.94)	55 (49–62)
2004: X	39	7	0.68 (0.47–0.83)	53 (50–57)
2005: I	46	8	0.96 (0.93–0.98)	59 (55–63)
2005: V	14	7	0.81 (0.59–0.94)	58 (53–63)
2005: X	37	7	0.95 (0.92–0.97)	55 (49–60)
2006: I	92	7	0.79 (0.68–0.87)	56 (54–59)
2006: V	20	8	0.87 (0.75–0.95)	53 (49–57)

n_a – no of abstracts; n_r – no of reviewers

Figure 1. Histogram of ISK abstract score for accepted and rejected abstracts.

Figure 2. Mean ISK abstract scores with 95% CI, by abstract acceptance for NOV scientific meetings.

Table 2.  Frequency and mean ISK abstract score for accepted (n = 229) and rejected (n = 190) abstracts

	Accepted				Rejected
			ISK score				ISK score
Year	n	%	mean	(95%CI)	n	%	mean	(95%CI)
2001	37	51	63	(60–66)	36	49	47	(42–53)
2002	18	75	63	(58–66)	6	25	46	(40–51)
2003	45	61	63	(60–65)	29	39	51	(47–55)
2004	14	36	60	(58–63)	25	64	49	(44–54)
2005	56	58	64	(62–66)	41	42	48	(42–53)
2006	59	53	62	(60–64)	53	47	49	(46–52)

Table

Scoring for abstracts of clinical studies (maximum 100 points)
Baseline score	+50
1. Problem description
clear	+5
vague	0
non-existent	–5
2. Design
prospective	+5
retrospective	0
questionnaire only	–5
not specified	–10
3. Control group
yes	0
multiple tests, one group	+5
matched	+5
randomized	+10
matched and randomized	+15
no control	–5
4. Material
unique	+5
adequate (size, length of follow-up)	0
insufficient (or not described)	–5
5. Methods
objective and valid	+10
described	0
not described	–5
6. Results
unique	+10
new and important	+5
existing knowledge	0
not important or incoherent	–5
not presented	–10
7. Conclusions
valid	0
not supported by results	–5
non-existent	–10

Table

Scoring for abstracts of experimental studies^a
Baseline score	+50
1. Problem description
important problem	+5
clear	0
vague	–5
2. Animal experiments only:
Design
Control group
yes	0
multiple, tests, same group	+5
matched	+5
randomized	+10
matched and randomized	+15
no control	–10
Methods
objective and valid	+10
described	0
not described	–10
3. Anatomy and biomechanics only:
Material (samples, specimens)
unique	+10
adequate	0
insufficient	–10
Methods
objective and valid	+15
well described (accuracy and technique	+5
described	0
not described	–10
4. Results
unique	+10
new and important	+5
existing knowledge	0
not important	–5
incoherent with methods or wrong	–10
not presented	–15
5. Conclusions
valid	+5
not supported by results	–5
non described	–10

^aLaboratory experiments, anatomy, biomechanics, animal studies