Adherence to Structured Risk Assessment Guidelines: Development and Preliminary Evaluation of an Adherence Scale for the START:AV

Figures & data

Table 1. Definitions of the fidelity dimensions applied to structured risk assessment practices.

Fidelity dimension	Definition in Implementation Research^a	Applied to structured risk assessment
Adherence	The extent to which the delivery of the intervention content is consistent with how it was designed or written.	Whether the evaluators completed the risk assessment instrument as it was designed and presented via the guidelines in a user manual (e.g., Did the evaluator rate all the items?).
Quality of Delivery	The extent to which a provider approaches a theoretical ideal in terms of delivering the intervention content (i.e., provider effectiveness).	The evaluator’s competence to conduct the assessment (e.g., Is the evaluator attending adequately to the rating criteria?).
Exposure/Dosage	The amount of intervention content received by participants in terms of number of sessions, duration, or intensity.	The number and frequency of risk assessments received by the intended population (e.g., Did individuals receive a risk assessment at the agreed times and occasions?).
Program Differentiation	Identification of an intervention’s unique features, without which it would not have the intended effect.	Identification of indispensable core features of the risk assessment instrument that are necessary for the instrument to reach its goals.
Participant Responsiveness	Participants’ engagement and involvement in the intervention.	How involved examinees are in the risk assessment (e.g., Is their own risk appraisal included?).

^aDefinitions are adapted from Dusenbury et al. (2003).

Table 2. START:AV Adherence Rating Scale.

	Insufficient (0)	Sufficient (1)	Good (2)
Information Gathering
•  Administration Data: All requested information is completed: youth’s name, date of birth, gender, assessment date, deadline, status, purpose and evaluator’s name.	Information is incomplete or incorrect.	-	All required information is complete.
•  Multiple Sources: Information is gathered through multiple sources.	Assessment is completed based on two or less file sources. Or used sources are not identified.	Assessment is completed based on multiple sources and/or collateral interviews. Youth is not interviewed.	Assessment is completed based on multiple sources, collateral interviews and interview with the youth.
START:AV items
•  Item Ratings: All strengths and vulnerabilities are rated (excl. case specific items and the item ‘culture’)	More than five strengths and/or five vulnerabilities are unrated.	Five strengths and/or five vulnerabilities are unrated.	Less than five strengths and/or five vulnerabilities are unrated.
•  Item Rating Rationale: For all strengths and vulnerabilities rated as moderate and high, a written rationale is provided or relevant anchors are marked.	More than 20% of the items with a moderate or high rating have no rationale or marked anchors.	Between 1% and 20% of items with a moderate or high rating have no rationale or marked anchors.	All items with a moderate and high rating have a rationale or marked anchors.
•  Key Strengths: At least two and maximum six key strengths are identified.	Less than 2 or more than 6 key strengths are selected.	2 key strengths are selected.	More than 2, but no more than 6 key strengths are selected.
•  Critical Vulnerabilities: At least two and maximum six Critical vulnerabilities are identified.	Less than 2 or more than 6 critical vulnerabilities are selected.	2 critical vulnerabilities are selected.	More than 2, but no more than 6 critical vulnerabilities are selected.
START:AV adverse outcomes
•  History Rating: For each adverse outcome, history is rated in a checkbox and explained in a text box. In forms without an ‘absent’ option, empty checkboxes can signify ‘not present’ or missing information. This can be inferred from the context. If not, rate ‘0’.	History is not rated for all adverse outcomes. History is clearly present (e.g., text box), but not rated.	History is rated for each adverse outcome, however not every history rating is explained with details.	For all adverse outcomes, history is rated and details are provided.
•  Risk Estimate: For each adverse outcome (incl. case specific if present), a risk estimate is provided.	Not all adverse outcomes have received a risk estimate.	-	For all adverse outcomes, a risk estimate is provided.
•  Risk Estimate Rationale: For each moderate and high risk adverse outcome, a rationale for the risk estimate is provided.	A rationale is not provided for all moderate and high risk estimates.	A global description is provided for moderate and high risk estimates. (Selecting “type” does not count as a description)	For each moderate and high risk estimates, a rationale is provided by linking the items to the adverse outcome.
•  T.H.R.E.A.T.: The tick boxes are completed.	Not all T.H.R.E.A.T.s are completed (Yes/No). A T.H.R.E.A.T. is rated but not all six conditions are met, or it is unclear.	-	All T.H.R.E.A.T.s are completed. When rated as present, all six conditions are met and illustrated.
•  Periods of Stability: The periods of stability text box is completed.	The text box is left empty.	-	Evaluator mentions and explains potential periods of stability. If not present, this is indicated (n/a).

Table 3. Interrater reliability of the START:AV Adherence Rating Scale items and total score.

Adherence item	Intraclass Correlation Coefficient
	Single Measure	95% CI	Average Measure	95% CI
1. Administration Data	.93	[.89,.95]	.96	[.94,.98]
2. Multiple Sources	.97	[.96, 98]	.99	[.98,.99]
3. Item Ratings	.75	[.64,.83]	.86	[.78,.91]
4. Item Rating Rationale	.84	[.76,.89]	.91	[.86,.94]
5. Key Strengths	.94	[.91,.96]	.97	[.95,.98]
6. Critical Vulnerabilities	.98	[.97,.99]	.99	[.99,.99]
7. History Rating	.89	[.83,.92]	.94	[.91,.96]
8. Risk Estimate	.92	[.88,.95]	.96	[.94,.97]
9. Risk Estimate Rationale	.81	[.72,.87]	.90	[.84,.93]
10. THREAT	.89	[.84,.93]	.94	[.91,.96]
11. Periods of Stability	1.00	[1.00, 1.00]	1.00	[1.00, 1.00]
Adherence Total Score	.96	[.93,.97]	.98	[.97,.99]

CI = confidence interval.

Table 4. Percentage of START:AV forms in each adherence category and median for the adherence items.

Adherence Item	Insufficient %	Sufficient %	Good %	Median
1. Administration Data	33.3	-	66.7	2
2. Multiple Sources	12.1	21.2	66.7	2
3. Item Ratings	4.3	1.0	94.7	2
4. Item Rating Rationale	10.2	37.5	52.3	2
5. Key Strengths	34.6	21.3	44.1	1
6. Critical Vulnerabilities	29.4	9.8	60.8	2
7. History Rating	28.8	14.7	56.5	2
8. Risk Estimate	8.5	-	91.5	2
9. Risk Estimate Rationale	32.7	45.1	22.2	1
10. THREAT	33.3	-	66.7	2
11. Periods of Stability	82.0	-	18.0	0

Adherence items 1, 8, 10 and 11 have no “sufficient” category.

Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237–256. https://doi.org/10.1093/her/18.2.237

 PubMed Web of Science ®Google Scholar