The Clinical Toxicology Recommendations Collaborative: purpose, organization, and methodology

Figures & data

Table 1. 9-point Likert scale for voting statements.

Vote	Meaning
1	The risks of the action greatly surpass or outweighs the potential benefits, and it should never be given in the circumstances described in the sentence
2	The risks of the action greatly surpass or outweighs the potential benefits, and it should almost never be given in the circumstance described in the sentence
3	The risks of the action likely surpasses or outweighs the potential benefits and it should generally not be given in the circumstance described in the sentence
4	The risks of the action probably surpasses or outweighs the potential benefits and it should generally not be given in the circumstance described in the sentence
5	The benefits of the action is equivalent to its risks (neutral)
6	The benefits of the action probably surpasses or outweighs the risks and it should probably be given in the circumstance described in the sentence
7	The benefits of the action surpass or outweighs the risks and it should generally be given in the circumstance described in the sentence
8	The benefits of the action greatly surpass or outweighs the risks and it should almost always be given in the circumstance described in the sentence
9	The benefits of the action greatly surpass or outweighs the risks and it should always be given in the circumstance described in the sentence
N/A	Not applicable: you don’t feel you can vote due to a lack of information to provide an evidence-base answer

Table 2. Sample disagreement index and vote calculations.

Panelists	Vote 1	Vote 2	Vote 3	Vote 4
1	9	9	9	9
2	8	8	8	9
3	7	7	7	9
4	6	6	6	6
5	9	9	4	6
6	9	9	2	2
7	8	8	4	6
8	7	7	7	9
9	6	6	6	6
10	9	9	9	1
11	8	8	8	8
12	8	8	8	2
13	6	6	6	6
14	7	6	7	1
15	9	9	9	3
16	9	9	5	5
17	5	5	5	1
18	6	6	6	1
19	7	7	5	1
20	4	1	4	4
21	9	9	9	9
Median	8	8	6	6
75th percentile	9	9	8	8
Lower limit of interpercentile range	7	6	5	2
Upper limit of interpercentile range	9	9	8	6
Interpercentile range	2	3	3	4
Interpercentile range center point	8	7.5	6.5	4
Asymmetry index	3	2.5	1.5	1
Interpercentile range adjusted for symmetry	6.85	6.1	4.6	3.85
Disagreement index (interpercentile range/ interpercentile range adjusted for symmetry)	0.29	0.49	0.65	1.04
Interpretation	Consensus around a strong recommendation	Consensus around a weak conditional recommendation	Consensus around a neutral recommendation	No consensus

In a hypothetical panel with 21 members (first column), the panellists vote on four different statements (second through fifth columns). The range of possible votes is shown on a Likert scale of 1 (strong disagreement with the statement) through 9 (strong agreement with the statement (Table 1). The midpoint of the Likert scale is 5. The authors first identify the median, 75th percentile, 25th percentile, and the interpercentile range center point (midway between the 75th and 25th percentiles). The asymmetry index is the absolute value of the distance between the interpercentile range center point and the scale center point of 5. The interpercentile range adjusted for asymmetry is calculated as 2.35 + (1.5* the asymmetry index). The disagreement index is then calculated as shown. Figure 1 shows how the median and the upper or lower quartile are used to provide the recommendation once the disagreement index is < 1.

Figure 1. Use of the disagreement index, the median of votes, and upper or lower quartile of the votes to arrive at the strength of recommendations made by the workgroup.

Table 3. Level of evidence to inform on the strength of recommendations.

Level of evidence (GRADE^a system)

Grade A: High level of evidence (The true effect lies close to our estimate of the effect).

Grade B: Moderate level of evidence (The true effect is likely to be close to our estimate of the effect, but there is a possibility that it is substantially different).

Grade C: Low level of evidence (The true effect may be substantially different from our estimate of the effect).

Grade D: Very low level of evidence (Our estimate of the effect is just a guess, and it is very likely that the true effect is substantially different from our estimate of the effect).

Strength of recommendation (consensus-based)

Level 1: Strong recommendation (The course of action is considered appropriate by the large majority of panel members with no major dissension. The panel is confident that the desirable effects of adherence to the recommendation outweigh the undesirable effects.)

Level 2: Weak/conditional recommendation (The course of action is considered appropriate by the majority of panel members but some degree of dissension exists among the panel. The desirable effects of adherence to the recommendation probably outweigh the undesirable effects.)

Neutral position: The course of action is neither preferred nor rejected by the majority of panel members; either due to a balance in the desirable and undesirable effects of adherence to the recommendation or due to major uncertainties to its evaluation.)

No recommendation: The panel members reached no agreement

^aGRADE: Grading of Recommendations, Assessment, Development, and Evaluations.