A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study

Figures & data

Fig. 1 The AFTE range of conclusions.

Fig. 11a Repeatability. No pooling of inconclusives (Report, p. 41).

Fig. 11b Repeatability. No pooling of inconclusives (Report, p. 41).

Fig. 12a Repeatability. Inconclusive categories pooled.

Table 12 Proportion of paired classifications in agreement and disagreement by the same examiner (repeatability) when (a) the three inconclusive categories are pooled, (b) ID and inconclusive-A are pooled and elimination and inconclusive-C are pooled -bullets.

	Proportion of paired agreements	Proportion of paired disagreements
Matching sets
(a) Pooling of inconclusives	83.4%	16.6%
(b) Pooling of ID with Inconclusive-A
and of Elimination with Inconclusive-C	85.5%	14.5%
Nonmatching sets
(a) Pooling of inconclusives	83.6%	16.4%
(b) Pooling of ID with Inconclusive-A
and of Elimination with Inconclusive-C	71.3%	28.7%

Note: Based on Report, Table XII, p. 42.

Table 13 Proportions of agreement: Bullet sets.

	Matches		Nonmatches
	Observed	Expected	Observed	Expected
Scoring	agreement	agreement	agreement	agreement
ID Inc-A Inc-B
Inc-C Elim	77.70%	63.70%	63.40%	55.80%
ID (Inc-A & Inc-B
& Inc-C) Elim	82.00%	66.40%	82.50%	77.00%
(ID & Inc-A) Inc-B
(Inc-C & Elim)	85.00%	75.70%	70.40%	63.90%

Note: From Report, Table XIII, p. 45.

Table 9. 1 Agreement table of observed percentages (%) for bullets non-matching.

	Identification	Inconclusive A	Inconclusive B	Inconclusive C	Elimination	Unsuitable
Identification	0.11	0.16	0.32	0.11	0.32	0
Inconclusive A	0	2.8	1.99	2.26	1.46	0
Inconclusive B	0.27	1.67	18.38	5.28	2.43	0.38
Inconclusive C	0.05	1.73	5.88	15.31	2.86	0.05
Elimination	0.05	1.08	1.89	3.56	27.71	0.22
Unsuitable	0	0	0.7	0.32	0.22	0.43

Table 9. 2 Agreement table of expected proportions (%)—Bullets non-matching.

	Identification	Inconclusive A	Inconclusive B	Inconclusive C	Elimination	Unsuitable
Identification	0	0.08	0.3	0.27	0.36	0.01
Inconclusive A	0.04	0.63	2.48	2.29	2.98	0.09
Inconclusive B	0.14	2.11	8.29	7.63	9.94	0.31
Inconclusive C	0.13	1.93	7.55	6.95	9.05	0.28
Elimination	0.17	2.57	10.06	9.26	12.07	0.37
Unsuitable	0.01	0.12	0.49	0.45	0.58	0.02

Table 9 Paired classifications by the same examiner (repeatability) for bullets.

Matching sets -Bullets
	Identification	Inconclusive A	Inconclusive B	Inconclusive C	Elimination	Unsuitable	Total
Identification	665	27	26	14	8	2	742
Inconclusive A	31	28	12	6	2	0	79
Inconclusive B	13	14	45	5	2	2	81
Inconclusive C	2	3	3	5	3	0	16
Elimination	8	7	3	2	13	0	33
Unsuitable	1	3	3	0	0	2	9
Total	720	82	92	32	28	6	960
Nonmatching sets -Bullets
	Identification	Inconclusive A	Inconclusive B	Inconclusive C	Elimination	Unsuitable	Total
Identification	2	3	6	2	6	0	19
Inconclusive A	0	52	37	42	27	0	158
Inconclusive B	5	31	341	98	45	7	527
Inconclusive C	1	32	109	284	53	1	480
Elimination	1	20	35	66	514	4	640
Unsuitable	0	0	13	6	4	8	31
Total	9	138	541	498	649	20	1855

Note: From Report, Table IX, p. 38.

Table

idn	incon-A	incon-B	Incon-C	elim	unsuit
1.02	8.51	28.41	25.88	34.51	1.67

Table

idn	incon-A	incon-B	Incon-C	elim	unsuit
0.48	7.44	29.16	26.84	35.00	1.08

Table

Value of Kappa	Level of Agreement
0–.20	None
.21–.39	Minimal
.40–.59	Weak
.60–.79	Moderate
.80–.90	Strong
Above.90	Almost Perfect

Fig. 12b Repeatability. Inconclusive categories pooled.

Table

	b	r	g	total first round
b	10	0	0	10
r	0	50	0	50
g	0	0	40	40
total second round	10	50	40	100

Table

	b	r	g	total first round
b	?	?	?	10
r	?	?	?	50
g	?	?	?	40
total second round	10	50	40	100

Table

	b	r	g	total first round
b	1	5	4	10
r	5	25	20	50
g	4	20	16	40
total second round	10	50	40	100

Table

	b	r	g	total first
				round
b	8+?	?	?	10
r	?	40+?	?	50
g	?	?	32+?	40
total second round	10	50	40	100

Table

	b	r	g	total first round
b	8	1	1	10
r	1	45	4	50
g	1	4	35	40
total second round	10	50	40	100

Table

	2nd Round
1st round	b	r	g	“marginal”
				1st round
b	0.01	0.05	0.04	0.1
r	0.05	0.25	0.2	0.5
g	0.04	0.2	0.16	0.4
“marginal” 2nd round	0.1	0.5	0.4	1

Table

	b	r	g
b	$\pi p_b+\gamma p_b^2$	$\gamma p_b{p}_r$	$\gamma p_b{p}_g$
R	$\gamma p_b{p}_r$	$\pi p_r+\gamma p_r^2$	$\gamma p_r{p}_g$
G	$\gamma p_b{p}_g$	$\gamma p_r{p}_g$	$\pi p_g+\gamma p_g^2$