Teen Well Check: an e-health prevention program for substance use, sexual assault, and sexual risk behaviors for adolescents in primary care

ABSTRACT

Objective: Adolescents are at risk for substance use, sexual assault, and sexual risk behaviours; however, to date no integrated prevention programmes address all three risk behaviours. The goal of this study was to evaluate the usability and acceptability of Teen Well Check, an e-health prevention programme targeting substance use, sexual assault, and sexual risk among adolescents in primary care settings.

Methods: The current study included content analysis of interviews with adolescents in primary care (aged 14–18; n = 25) in the intervention development process, followed by usability and acceptability testing with qualitative interviews among adolescents in primary care (aged 14–18; n = 10) and pediatric primary care providers (n = 11) in the intervention refinement process. All data were collected in the Southeastern U.S.

Results: Feedback on Teen Well Check addressed content, engagement and interaction, language and tone, aesthetics, logistics, inclusivity, parent/guardian-related topics, and the application of personal stories. Overall, providers reported they would be likely to use this intervention (5.1 out of 7.0) and recommend it to adolescents (5.4 out of 7.0).

Conclusions: These findings suggest preliminary usability and acceptability of Teen Well Check. A randomized clinical trial is needed to assess efficacy.

Highlights

• Adolescents are at risk for substance use, sexual assault, and sexual risk behaviours.

• The goal of this study was to evaluate the usability and acceptability of Teen Well Check, an e-health prevention programme targeting substance use, sexual assault, and sexual risk among adolescents in primary care settings.

• Providers and adolescents rated Teen Well Check as usable and acceptable, and providers indicated that they would recommend it to their adolescent patients.

Objetivo: Los adolescentes están en riesgo de uso de substancias, agresión sexual y conductas sexuales de riesgo; sin embargo, hasta la fecha, no existen programas de prevención integrados que aborden estas tres conductas de riesgo. El objetivo de este estudio fue evaluar la utilidad y aceptabilidad de Chequeo del Bienestar del Adolescente (Teen Well Check), un programa electrónico de salud centrado en el uso de substancia, agresión sexual y riesgo sexual entre adolescentes en contextos de atención primaria.

Métodos: El presente estudio incluyó análisis de contenido de entrevistas con adolescentes en atención primaria (edades de 14 a 18 años; n = 25) en el proceso de desarrollo de la intervención, seguido por la evaluación de la utilidad y aceptabilidad con entrevistas cualitativas con adolescentes en atención primaria (edades de 14 a 18 años; n = 10) y proveedores de atención primaria a nivel pediátrico (n = 11) en el proceso de refinamiento de la intervención. Todos los datos fueron recolectados en el sudeste de los Estados Unidos.

Resultados: La retroalimentación del Teen Well Check abordó contenido, participación e interacción, lenguaje y tono, estética, logística, inclusión, temas relacionados con los padres o cuidadores, y la aplicación de historias personales. En general, los proveedores reportaron que ellos probablemente usarían esta intervención (5.1 de 7.0) y la recomendarían a los adolescentes (5.4 de 7.0).

Conclusiones: Estos hallazgos sugieren la utilidad y aceptabilidad preliminar de Teen Well Check. Un ensayo clínico aleatorizado es necesario para evaluar la eficacia.

目的：青少年面临物质使用、性侵犯和危险性行为的风险； 然而，迄今为止，还没有针对所有三种风险行为的综合预防方案。 本研究旨在评估青少年健康检查的可用性和可接受性，它是一项针对初级医疗环境中青少年物质使用、性侵犯和性风险的电子健康预防计划。

方法：本研究包括在干预开发过程中对初级医疗青少年（14–18 岁；n = 25）进行访谈的内容分析，然后对初级医疗青少年（14–18岁；n = 25）（14–18 岁；n  = 10) 以及儿科初级医疗提供者 (n = 11) 在干预细化过程中的定性访谈进行可用性和可接受性检验。 所有数据均在美国东南部收集。

结果：对青少年健康检查的反馈涉及内容、参与和互动、语言和语气、审美、后勤、包容性、父母/监护人相关主题以及个人故事的应用。 总体而言，提供者报告说他们可能会使用这种干预措施（7.0 分中得5.1 分）并推荐给青少年（7.0 分中得 5.4 分）。 提供者将干预描述为对他们的患者‘有帮助’（7.0 分中得 5.2 分）。

结论：这些发现表明了青少年健康检查的初步可用性和可接受性。 需要进行随机临床试验来评估疗效。

KEYWORDS:

• Prevention
• adolescents
• e-health
• drug use
• sexual assault
• sexual health

PALABRAS CLAVE:

• Prevención
• adolescentes
• salud electrónica (e-salud)
• consumo de drogas
• agresión sexual
• salud sexual

关键词:

• 预防
• 青少年
• 电子健康
• 药物使用
• 性侵犯
• 性健康

Substance use, sexual assault, and sexual risk behaviours among adolescents are common and interrelated (Johnston et al., 2021; Scott-Sheldon et al., 2016; Smith et al., 2018). Among adolescents surveyed in the United States during 2018, past year substance use was 38.3% for alcohol, 24.6% for cannabis, 30.7% for any vaping, and 9.2% for illicit drugs other than cannabis (Johnston et al., 2021). Substance use and sexual behaviour often co-occur, with 22.4% of sexually active adolescents aged 14–18 years using substances before their most recent sexual encounter (Johnston et al., 2016). Impairments in sexual decision-making due to substance use can lead to a range of consequences, including sexual risk behaviours and non-consensual sex. Adolescents are disproportionally affected by the consequences of sexual assault and sexual risk behaviours, being more likely to contract sexually transmitted infections (STIs; Kreisel et al., 2021) be sexually victimized (Black et al., 2011), and experience hardship due to unwanted pregnancy (Noll et al., 2019). Prevalence of these risk behaviours among youth has increased since the onset of the COVID-19 pandemic (Dumas et al., 2020; Masonbrink et al., 2022). Prevention of these three interrelated major public health concerns would significantly reduce mental and medical health burden.

Perceptions of peer behaviour and attitudes, or perceived social norms, are associated with alcohol use, drug use, sexual assault, and sexual risk behaviours among adolescents and young adults (Hoxmeier et al., 2018; Kantawong et al., 2021; Pedersen et al., 2017). Communication skills including how to talk about sex and sexual consent, as well as communication with parents and providers about risk behaviours, are associated with decreased alcohol use, drug use, sexual assault, and sexual risk behaviours (Hops et al., 2011). Personalized normative feedback and communication skills practice may be an effective cross-cutting strategy to reduce and prevent risk for alcohol and drug use, sexual assault, and sexual risk behaviours among adolescents.

Many adolescents seek routine preventative healthcare in primary care settings. Recent health reforms have improved population health outcomes and primary care services (Hofer et al., 2011). e-Health screening and brief interventions allow standardized screening for adolescents prior to meeting with the primary care provider. Adolescents also prefer technology-based screening (Gibson et al., 2021) compared to screening conducted by a provider. A systematic review and meta-analysis of eHealth interventions with adolescents to prevent risk behaviours found beneficial effects across a range of behaviours (Champion et al., 2019). This manuscript describes the development and acceptability testing of an e-Health integrated prevention programme for adolescent substance use, sexual assault, and sexual risk behaviours in primary care settings using the Intervention Mapping framework (Bartholomew Eldredge et al., 2016). The aim of this project was to develop an intervention that could be disseminated into primary care settings without adding burden to primary care providers.

1. Methods

1.2. Participants and procedures

All study procedures (see Figure 1) were approved by the university IRB and were completed between April 2018 and October 2021. All data were collected in the Southeastern USA.

Figure 1. Study flow.

The intervention was developed using the Intervention Mapping framework (Bartholomew Eldredge et al., 2016) and followed practices consistent with the Template for Intervention Description and Replication (TIDieR) checklist (see Table 1). For a description of the target outcomes and theoretical components, see Table 1. Theory-based processes and qualitative interviews with adolescents (Intervention Development Phase described below) were used to develop the intervention. Using the Intervention Mapping Framework, we translated methods and strategies into an organized programme, consulted with intended participants and stakeholders, prepared programme materials and piloted programme materials for feedback on the acceptability of the programme from providers and adolescent patients (Intervention Refinement Phase described below). In the intervention development process, we prioritized using theory-based methods and content that were acceptable to providers and adolescent patients that would not add burden to primary care providers and disrupt clinic flow. Stakeholders in this process were primary care providers and they gave in-depth feedback on Teen Well Check during the Intervention Refinement phase.

Table 1. Template for Intervention Description and Replication (TIDieR) checklist.

	Teen Well Check TIDierR checklist
Why:	A brief, technology-based integrated prevention for adolescents focused on substance use, sexual assault, and sexual risk in primary care settings has the potential to prevent related risk behaviours in a brief format that can be delivered as part of a wellness visit that does not add burden to primary care providers.
What (material):	See Table 2 for a description of intervention components and Figure 2 for examples of the content included in the technology-based intervention.
What (procedures):	Teen Well Check can be delivered to adolescents aged 14–18 visiting a primary care clinic and can be delivered in the waiting room, while waiting in a clinic room, or after the visit.
Who provided:	Teen well check is a self-led programme and does not require any particular training or person to deliver the programme. However, it was developed to be provided in the context of a primary care visit.
How (mode of delivery):	Teen Well Check is a technology-based programme that is delivered through a web-app. The web-app can be accessed via a tablet, smart phone, or computer.
Where:	Pediatric primary care clinics. If delivered in the waiting room or a clinic room, the clinics would need a tablet, smart phone, or computer that is connected to internet. If delivered after the primary care visit, the patient would need access to a tablet, smart phone, or computer that is connected to internet.
When and how much:	Teen Well Check is a single session programme. It takes approximately 5–15 min to complete.
Tailoring:	The current iteration of Teen Well Check is tailored by age, gender, and participant report of substance use. Participants are provided with age and gender tailored normative data throughout the intervention. Further, if participants report no substance use, they are able to choose which substance they want to learn about. If they report using one substance, they are given information about that substance. If they report using multiple substances, they are able to choose which substance they want to learn about out of those they have used. Given that it is a technology-based programme it could be tailored more specifically in future iterations.
Modification:	Yes, the purpose of this study was to modify the intervention based on feedback from providers and adolescents. Please see Figure 2 for examples of how the sample content changed after feedback.
How well (planned):	In the current study, participants reviewed the entire programme because an interview took place after viewing the intervention content. Future work needs to assess fidelity to the programme when delivered as part of a feasibility trial. Therefore, adherence was not assessed as an outcome in the current study.
How well (actual):	Adherence was not assessed as an outcome in the current study.

1.2.1. Intervention development phase

Adolescents aged 14–18 years old (n = 25), attending a wellness visit at their primary care doctor were approached in the waiting room or patient room by research staff to complete an eligibility screen for the study. See Table 2 for participant information. All procedures were conducted at academic medical centres that primarily served low-income families insured by Medicaid. The eligibility screen assessed if adolescents met the following eligibility criteria: between 14 and 18 years old, could read and understand English, did not have a developmental disability that would prevent them from understanding the programme contents, and reported knowing a peer who has ever used substances. Eligible individuals then consented (parents) and assented (adolescents) to the study procedures. Adolescents completed a questionnaire on a tablet, interview to provide feedback on the intervention development phase of Teen Well Check (see Figure 2 and below for more information) using screenshot mockups shown to adolescents on a tablet by research staff, and questionnaire about the intervention on a tablet. All study procedures after the consent process were conducted in a private room in the clinic without the parent present. Participants received a $20 gift card for participation.

Figure 2. Sample content from Teen Well Check across each phase of the study.

Table 2. Theory- and evidence-based components in Teen Well Check.

Target outcome	Theoretical framework	Prevention components
Substance use	Motivational Interviewing (Naar-King & Suarez, 2011)	Screening, reinforcement for non-use, self-identified consequences of use (Dimeff et al., 1999; Naar-King & Suarez, 2011; National Institute on Alcohol Abuse and Alcoholism, 2011)
	Social Norms Theory (Borsari & Carey, 2001; Collins et al., 1985; Lewis & Neighbors, 2004; Perkins, 2002)	Correcting misperceptions of perceived and actual peer norms of alcohol and drug use
	Ecological Framework (Bronfenbrenner, 1979)	Increase communication skills with peers (including romantic and sexual partners), parents, and medical providers regarding use
	Negative Reinforcement Theory (Skinner, 1958)	Psychoeducation on traumatic stress symptoms and use of substances to cope with traumatic stress symptoms, encouraging active coping strategies
Substance use before sex	Sex-Related Substance Use Expectancies (e.g. Alcohol and Marijuana Expectancy Theory) (Hendershot et al., 2010)	Interactive feedback regarding sex-related substance use expectancies (Gilmore et al., 2022)
	Impairment Theories (e.g. Alcohol Myopia Theory) (Steele & Josephs, 1990)	Interactive feedback applying alcohol myopia theory to sexual decision-making (Gilmore et al., 2022)
Sexual risk behaviours	Social Norms Theory	Correcting misperceptions of perceived and actual peer norms of sexual risk behaviours (Lewis et al., 2014)
	Skills Deficit Model	Interactive education to increase condom use, condom use self-efficacy, and sexual communication skills
Sexual assault victimization	Ecological Framework/Cognitive Mediational Model (Gidycz et al., 2006; Nurius & Norris, 1996; Ullman, 2014)	Increase sexual communication skills, sexual assault risk perception skills, and effective use of resistance strategies through scenarios (Gilmore et al., 2022)
Sexual assault bystander intervention	Social Norms Theory/Bystander Education (Fischer et al., 2011; Jouriles et al., 2018; Labhardt et al., 2017; Latané & Darley, 1970; Latané & Nida, 1981; Latané & Rodin, 1969)	Interactive bystander education, psychoeducation on substance use and bystander intervention (Gilmore et al., 2022)

1.2.2. Intervention refinement phase

Adolescents aged 14–18 years old (n = 10), completed the same procedure process as the Intervention Development phase but viewed the entire programmed Teen Well Check programme (see Figure 2 and below for more information) that was programmed in a tablet-based application. See Table 2 for participant information. There was no overlap in adolescents in the samples.

Further, we completed usability and acceptability testing via interviews among pediatric primary care providers in community clinics (n = 11). Providers were shown screenshots of the programme during the interview and were compensated via a $100 gift card drawing.

1.3. Teen Well Check development using iterative, end-user focused design

Teen Well Check contains three modules created for this intervention specific to substance use, sexual risk behaviour, and sexual assault, presented in a linear fashion using a motivational interviewing approach. The content of the intervention was designed based on existing literature and published theory as described below and outlined in Table 2.

1.3.1. Substance use module

Participants received personalized feedback on perceived and actual age- and gender-specific substance use, psychoeducation regarding the effect of substance use on brain development, their own consequences experienced, and psychoeducation related to the substance of choice (1. Vaping/cigarettes/Juul; 2. Alcohol; 3. Cannabis/marijuana; 4. Prescription opioids; 5. Other drugs). If participants only used one substance, they learned about that substance. If they used multiple substances, they chose which substance they had used that they wanted to learn about. If participants had not used substances, they were provided an option to learn about any substance. Psychoeducation also included a short video developed by NIDA about brain anatomy, physiology, and effects of substance use (National Institute on Drug Abuse [NIDA/NIH], 2019).

1.3.2. Sexual risk module

Participants were asked questions and provided feedback about how many adolescents their age had been tested for STIs and how many used substances prior to engaging in sexual behaviour. Psychoeducation focused on condom use and how substances can impact sexual communication.

1.3.3. Sexual assault module

Scenarios of potential sexual assault situations were provided where participants could learn how to respond as a potential victim, perpetrator, or bystander. Information regarding sexual consent was also provided, including information about how substances can impact sexual consent.

1.3.4. Teen Well Check intervention development

The intervention development phase (see Figure 2) included ten mock-up designs that were displayed to participants on a tablet. Mock-up designs were developed using theory-based cross-cutting targets identified by the research team. After receiving feedback on the intervention development phase mock-ups, interviews were analyzed. Once results were identified and thematic saturation was achieved, adaptations were made to the content by the research team.

1.3.5. Teen Well Check intervention refinement

Teen Well Check was optimized for use on a tablet to be delivered in primary care waiting rooms. It was presented to teens and providers, where interviews were determined to be completed after reaching thematic saturation. Adolescent feedback was used to make adaptations to the content by the research team during the final iteration, which was programmed into a web-based prevention programme (see Figure 2). In each stage, all adaptations requested were made unless there was disagreement among participants or if it was not feasible within the scope of the study.

1.3.6. Teen Well Check logistics

Before the COVID-19 pandemic, teen wellness visits usually lasted 30 min and intake paperwork (including substance use screening forms) was completed before arrival or in the waiting room before the appointment started. The initial concept of Teen Well Check was for it to be delivered in the waiting room in a single session of 5–15 min. Due to the clinic flow shift that occurred during the pandemic, patients waited in their cars or outside of the clinic rather than in waiting rooms as was typical pre-COVID. For this reason, combined with the concern about sanitizing requirements for tablets, Teen Well Check was programmed to be accessible remotely via a personal device.

1.4. Measures

Interview. Adolescents and providers completed a semi-structured interview ranging from 30 to 60 min to provide feedback about Teen Well Check, including aesthetics, content, and interest in using the intervention.

Acceptability and usability were assessed among adolescents and providers using the Post Study System Usability Questionnaire (PSSUQ) (Lewis, 2002) and questions developed by the research team. The PSSUQ is a 19-item instrument that measures satisfaction with a computer system (Bartholomew Eldredge et al., 2016) with answer choices ranging from ‘Strongly Agree’ (1) to ‘Strongly Disagree’ (7). Lower scores indicate more acceptability and usability. Providers were also asked the likelihood of using and recommending Teen Well Check ranging from ‘Very Unlikely’ (1) to ‘Very Likely’ (7). Mean scores will be used. (Table 3).

Table 3. Characteristics of primary care providers (n = 11) and adolescents 14–18 years (n = 35).

Characteristic	Providers		Adolescents (Intervention Development)		Adolescents (Intervention Refinement)
	N	%	N	%	N	%
Gender
Female	11	100%	14	56%	8	80%
Male	0	0%	9	36%	2	20%
Other (Genderfluid, Two Spirit)	0	0%	2	8%	0	0%
Age			16.3 (M)	1.37 (SD)	16.7 (M)	1.49 (SD)
Race
White/Caucasian			9	36%	6	60%
Black/African American			11	44%	1	10%
Asian American/Pacific Islander			0	0%	1	10%
Biracial			2	8%	1	10%
Other/Mixed			3	12%	1	10%
Ethnicity
Hispanic/Latinx			2	8%	1	10%
Not Hispanic/Latinx			23	92%	9	90%
Sexuality
Straight/Heterosexual			15	60%	5	50%
Bisexual			4	16%	3	40%
Gay/Lesbian			4	16%	0	0%
Questioning					2	20%
Other (Pansexual)			2	8%	0	0%
Tablet Access			24	96%	9	90%
Past year alcohol use			3*	12%*	4	40%
Past year tobacco use			N/A*	N/A*	5	50%
Past year drug use			1*	4%*	7	70%

*Note: During Aim One, participants were not asked about smoking behaviours or past year of alcohol or drug use. These results for Aim One reflect past month alcohol and drug use.

1.5. Qualitative data analysis

Interviews were content analyzed using an iterative team-based approach led by a PhD-level researcher. The researcher reviewed each transcript and developed a list of topics that emerged both deductively from the interview guide and inductively from participants’ responses. Using those topic lists as a preliminary codebook and NVivo (QSR International Pty Ltd, 2018), two research staff members separately coded (White et al., 2022) the transcripts, compared and refined codes for each transcript, and reconciled their coding until they had 100% agreement (i.e. reviewed coding together in real-time and came to agreement). Coders developed analytic memos for each code, noting major findings and important quotations. The researcher used those analytic memos, reports from NVivo (ex: code co-occurrence, transcript coverage), and the coded quotations to identify the most salient results.

2. Results

2.1. Intervention development phase of Teen Well Check: adolescent feedback

2.1.1. Aesthetics

Participants provided feedback on how to improve aesthetics of pictures, animations, and videos (see Quote 1 in Table 4). Formatting feedback was provided to improve aesthetic acceptability of the intervention, including commenting on the colour scheme and font. Participants suggested adding more visual components and images to enhance usability throughout the programme. Participants commented on the importance of using images and language that were inclusive and representative of diverse identities (see Table 4 for sample quotes).

Table 4. Sample feedback quotes on phase of Teen Well Check.

Adolescent interviews				Provider interviews
Intervention development results	Sample quotes	Intervention refinement results	Sample quotes	Intervention refinement results	Sample quotes
Aesthetics	(1) ‘Well, color-wise, there are like, or from what I’ve heard, there are colors that attract the human eye more. So, like color-wise, making it, a little more colorful, maybe even the font wise. It's some, it's trying to reach each teenage group, so it's like, if I see color, like if see the love right there and it's in red, it's like a pop out.’	Engagement and Interaction	(5) ‘I actually really liked that because it actually helps people learn about your brain. And I didn't even know that some things are certain things. So, I think that's really cool … I like how you added a video too. That's like, I like videos.’	Engagement and Interaction	(6) ‘I like the way you guys asked questions that they had to kind of guess on and then you gave them the correct response. So I like that it was interactive.’
Alcohol and Substance Use	(2) ‘Some people don't really know about the brain but, this just tells you about it basically and how you could stop doing that because this is how it could affect you like you’re just thinking about it, like oh people told me about lungs, smoking and that but then your brain is also slowly functioning because of you doing that. Yeah, I like that.’	Language and Tone	(7) ‘I feel like it's good. I don't think it's very criticizing of people. So, it's very open-ended. And I like how you already included the alcohol at the party. So, it's not like, oh, you should just avoid all risky situations to begin with and like – I like how it automatically – I mean, they’ll understands that we do have – people have sex. And it's not just, oh, you just use abstinence, like it welcomes the idea that you are allowed to do things that like you’re allowed to drink and you’re allowed to, like, have fun and like, engage in sexual activity as long as you’re safe. So, it's kind of acknowledging maybe like, yes, people do instead of ignoring like, oh, you should just not do it until later.’	Language and Tone	(8) ‘It sounds harsh. I’m having trouble articulating but like almost like judgy … So anything that has somewhat like all or nothing language, I think unconsciously can turn some kids off and … sort of missed an opportunity. So just subtle things like the word “forever,” which is true, but … You know, maybe that's too darkening … When you do these kind of screening things. The more that you feel you can be asked without judgment. I think the more it lands.’
Healthy Relationships (Sexual Risk)	(3) ‘When I took sex ed and like, I think it was eighth grade maybe, I saw this video about like how to put on a condom and it was like this cartoon thingy –– it was teaching us how to put condoms on and he was like putting it on himself and like sliding it down, it was kind of weird but it was like funny and education at the same time. So, maybe [include] something like that.’	Inclusivity	(9) ‘There's a lot more identities coming out now. So, just to be a little more inclusive, because a lot of people [are] like non-binary.’	Inclusivity	(10) ‘I feel like you guys have tried to have graphics that are like representing diverse individuals. But I think you could just, like, make sure you’re looking at those again because like this page, it looks like a lot of Caucasian. Like five of the six people, at least from the view that I can see what Caucasian and teams. Want to see themselves represented … yeah but being sure to have like diverse body types and skin colors, etc … Not same sex couples, but I think all those are important considerations.’
Consent (Sexual Assault)	(4) ‘Like, I think with a girl you could say … Like, I don't want you to do that or like, no you can't do that, sorry. Or don't even apologize because it's your body. You could say something like, that's making me uncomfortable, can you stop that. I like you, but I don't want to do that.’	Aesthetics and Images	(11) ‘Don't use people pictures.’	Aesthetics and Images	(12) ‘You know, thinking teenager mindset, “hokey” stock pictures are their least favorite in the world. There's no way to avoid that.’
		Personal Background and Use of Stories	(13) ‘I think I know most about drugs because, my mom did drugs with my dad, so that's why I live with my grandparents. And I’ve seen what it does to you, so I stay away … I’ve seen what other kinds don't and they don't know what actually happens. And they can't feel what I’ve gone through … it's not real to them, they just don't understand … If they’re doing it, they aren't going to stop because we’ve already been taught not to do it. Unless something dramatic happens … I don't know maybe put in some scary pictures … what it looks like after you use this type of stuff. And what I think really gets to people are like having other people's stories be heard. Like in middle school we had this guy come in and he was telling us about his son and like I don't know – I feel like you can't really get personal with just some slides and stuff.’	Personal Background and Use of Stories	(14) ‘Some of this content is like a lot of fact-based content. I don't know if there's any role of like … More of a like personal story or anecdote. Like that's good content, maybe just might feel a little more applicable if it wasn't Just numbers and facts, but like maybe what happened to someone who use this or something like that. So it felt like maybe a little bit more relatable.’
				Logistics and Information Sharing	(15) ‘If you were to maybe expand into take-home because I do understand time is an issue so that would be very good. But if it was like a take-it-home and do it separately, I would probably include all of them [modules] just for educational purposes.’
		Parent/guardian-related Topics	(16) ‘So having some important resources like this, we just don't want it to be something that's going to cause the person holding this tablet to like, click out of it or shut down and be like, “I don't want to see this” or even something that would make them uncomfortable with it if they’re in the waiting room … Especially like, if it's an underage person, their parent is going to probably be like, right there and watching … So that's also a big issue, especially for me. So like, if I was a kid and say, I’m like, 14, and my mom's right there, so many questions I’m not going to … I wouldn't answer honestly if my parents in, they are sitting right there.’	Parent/guardian-related Topics	(17) ‘I think most of our parents will be pretty positive about it. I think a lot of my parents are not naive to think that their kids are exposed or engaging and some of these behaviors … We have a smaller section of patients and families that would be very uncomfortable. They’re very uncomfortable with just alone time with us in the room. And they get very nervous and want to know exactly what we’re talking about. So I think those families would have difficulty with it … But I think for the majority of our families … they would welcome any chance to educate their kids and try and save them from risky behavior.’

2.1.2. Alcohol & substance use

Participants also provided feedback on content-specific aspects of the intervention. Specific to substance use, participants liked the alcohol-related content and responded positively to brain-related substance use content (see Quote 2 in Table 4). Participants also suggested expanding information about substance use. Participants requested more content on the relation between substance use and the brain, wanted definitions or content to be clarified, and wanted more information on the consequences of substance use (see Table 4).

2.1.3. Sexual risk behaviours

Overall, participants had positive reactions to the sexual risk behaviour content. Participants suggested including content on the prevalence of STIs and the consequences of sexual risk behaviours without protection or with multiple partners. Participants reported that it would be useful to include a video on using sexual protection such as a condom properly (see Quote 3 in Table 4).

2.1.4. Sexual assault

Participants desired additional strategies to prevent sexual assault. Participants also suggested adding more scenarios from the perspectives of the victim, perpetrator, and bystander. Participants provided feedback on sections related to consent and wanted consent definitions with examples of consensual and non-consensual sexual activity. Participants also expressed the need for language around how to ask for consent, sexual refusal, and the importance of ongoing consent (see Quote 4 in Table 4).

2.2. Intervention refinement phase of Teen Well Check: adolescent and provider feedback

2.2.1. Engagement and interaction

Providers and adolescents indicated that Teen Well Check was engaging and interactive through its inclusion of videos, interactive questions and scenarios, and personalized tailoring to adolescents’ pre-survey answers and preferences (see Quote 5 in Table 4). Providers and teens universally had positive comments about including the short video developed by NIDA about brain anatomy, physiology, and effects of substance use (National Institute on Drug Abuse [NIDA/NIH], 2019).

Providers and adolescents wanted more interactive components of Teen Well Check. Specifically, responses indicated that they would like to see more interactive bystander scenarios (e.g. more ‘clicking,’ more videos, and more ‘quizzes’ to test their knowledge; see Quote 6 in Table 4).

2.2.2. Language and tone

Providers and adolescents emphasized the need for brevity, clarity, and not including too much information or text on the screen. Providers and adolescents both appreciated clear definitions and intentional diction. Generally, providers and adolescents agreed the app did and should continue to use a motivational interviewing approach rather than ‘scare tactics’ and should ‘meet people where they are’ rather than ‘being judgy’ or presenting ‘all or nothing’ scenarios (see Quotes 7 and 8 in Table 4).

2.2.3. Inclusivity

Providers and adolescents encouraged more inclusivity throughout the Teen Well Check app, particularly for trans, non-binary, or those with gender non-conforming identities, as well as inclusion of racial and ethnic minorities. Participants suggested the inclusion of same-sex couples, images of non-binary people, and avoiding binary language such as ‘boy–girl’ or ‘he-she.’ Some providers and adolescents pointed out that the sexual assault scenarios all positioned boys as perpetrators and girls as victims and suggested having scenarios with boys as victims (see Quotes 9 and 10 in Table 4).

2.2.4. Aesthetics and images

Providers and adolescents mostly disliked the stock photos used throughout Teen Well Check, preferring the comic book style images used within the sexual assault module. One participant noted, ‘The circle full of teenagers, it just— it's like a little cliché,’ as did another, stating, ‘Don't use people pictures.’ In fact, some providers predicted that adolescents would find the photos of teens to be ‘hokey’ while also suggesting adolescents would respond well to the comic style (see Quotes 11 and 12 in Table 4). Providers liked Teen Well Check's bright colours, and most adolescents said they enjoy the ‘very bright’ colours ‘which really attracts people's attention’. They noted that they wanted the programme to be ‘aesthetically pleasing’ and ‘cute’.

2.2.5. Personal background and the use of stories

Several providers and adolescents mentioned the importance of using stories and anecdotes to help users understand and engage more deeply with the material (see Quotes 13 and 14 in Table 4).

2.2.6. Logistics and information sharing

Providers expressed concerns about how the 15-minute Teen Well Check intervention would be delivered in the waiting room, including lack of sufficient time and privacy (e.g. from guardians and other patients), difficulty ‘keeping a teenager's focus’ in the context of the pandemic clinic flow. Several participants suggested the intervention should be made available online, allowing participants to complete in their own time from a convenient and private location (see Quote 15 in Table 4). Providers also emphasized the importance of sharing resources with adolescents, connecting them to necessary referrals, and sharing information with the parents to offer appropriate care.

2.2.7. Provider impressions of parent/guardian-related topics

When researchers asked how guardians would respond to Teen Well Check, providers described a variety of expected responses, anticipating that some guardians would be supportive, with others being uncomfortable, particularly with ‘the sex part.’ Providers emphasized they ask their adolescent patients drug, alcohol, and sex questions, but it is done during the confidential history portion of the exam, as allowed by state laws. Adolescents similarly emphasized the importance of resources like Teen Well Check but expressed concerns about confidentiality and privacy (see Table 4).

2.3. Acceptability and usability: adolescents and providers

Most providers were very interested in using a prevention programme like Teen Well Check in their practice, especially if the necessary technology is provided. This high level of interest was reflected in our survey data (see Table 5). When asked how likely they would be to (1) use this intervention and (2) recommend it to 14–18-year-olds, average provider responses indicated that providers would be somewhat likely to use the intervention in their clinical practice and recommend that adolescents use it. Providers were somewhat concerned about using a tablet-based intervention in the waiting room given the new clinic flow due to the pandemic, and recommended an online format. Therefore, the final version of Teen Well Check was adapted to an online format to be used at the clinic or after teens’ appointments based on clinic flow.

Table 5. Usability and acceptability of the Teen Well Check tablet-based intervention among primary care providers (n = 11) of adolescents 14–18 years in the Intervention Refinement process (n = 10).

Survey Item	Provider Average Score; Mean (SD)	Adolescent Average Score; Mean (SD)	Scale
Post-Study System Usability Questionnaire	1.84 (1.10)	1.24 (0.35)	1=Strongly Agree (Usable)	5=Strongly Disagree (Not Usable)
How likely would you be to use this tablet-based intervention in your clinical practice?	5.09 (1.76)		1=Very Unlikely	7 =Very Likely
How likely would you be to recommend that teens aged 14–18 use this tablet-based intervention?	5.36 (1.50)		1=Very Unlikely	7 =Very Likely

In the interviews, most providers indicated that Teen Well Check was appropriate for 14–18-year-olds. Other providers felt the topics needed to be addressed at a younger age (e.g. 12 years old). Providers and adolescents ‘strongly agreed’ or ‘agreed’ that Teen Well Check was highly usable as assessed by the PSSUQ (M_provider = 1.84; M_adolescents = 1.24; see Table 5).

3. Discussion

The current study assessed the usability and acceptability of Teen Well Check, an e-Health prevention programme targeting substance use, sexual assault, and sexual risk among teens in primary care settings, which provides an ideal window of opportunity for identification and prevention education for adolescents. Though the scientific premise for integrating prevention draws from previous work which outlines the interrelated nature of these health risk behaviours (George & Stoner, 2000; Hendershot & George, 2007; Testa & Livingston, 2009), no preventive work has been conducted to date targeting and integrating these three co-occurring risk behaviours among adolescents. Overall, teens and providers found the programme to be acceptable and useable, and providers indicated that they would be somewhat likely to use it in their clinical practice and recommend to adolescents. Teens and providers gave suggestions throughout the intervention's development, which were integrated into the final version of the programme.

When assessing feasibility of implementing Teen Well Check within a primary care setting, providers indicated that changes in clinic flow due to the direct result of the COVID-19 pandemic, a tablet-based programme delivered and completed in clinic waiting rooms may not be feasible. Therefore, the final version was accessible remotely via a personal device to maximize the connectivity of the prevention programme to the healthcare setting, given that is where adolescents learn about the programme, without taking away from routine healthcare. This approach could eventually allow for a very scalable implementation strategy if the programme is found to be efficacious because it can be made broadly accessible nationwide rather than based in specific clinics.

Adolescent participants indicated that improvements in usability and engagement may be achieved through gamifying Teen Well Check. Although this approach to intervention programming would be innovative and potentially useful, it was not feasible to use gamification due to development budget limitations during this study. Future research should consider including gamification in prevention programmes, as well as videos, comics, or quotes to integrate personal stories to enhance prevention, a strategy participant in the present study also suggested.

3.1. Limitations

To protect anonymity, the current interface does not integrate with medical records; however, to implement with pediatric primary care clinics nationwide, it may be useful for clinic buy-in to create a provider dashboard and to integrate with an electronic medical record. This would require significant funds due to the complexity of software needed to integrate with electronic medical record systems. Further, participant interactions with the programme were not recorded due to programming limitations in initial versions, which was addressed in the final version of the programme. A feasibility trial is needed to understand other field notes within one or more primary care settings to learn more about use, process, and how the providers would interface with the programme. If Teen Well Check is found to be feasible and efficacious, there is potential to expand this intervention to be tailored for specific groups of adolescents including tailoring based on race/ethnicity, sexual orientation, and age. Given that this is a prevention programme, it may be possible to tailor to a younger age as well.

4. Conclusions

The current study presents findings about the development and acceptability of Teen Well Check. Future research is required to understand if Teen Well Check effectively prevents substance use, sexual assault, and sexual risk among teens. Overall, Teen Well Check was acceptable to teens and providers, and prevention programmes targeting substance use, sexual assault, and sexual risk may be useful within primary care settings. More work is needed to assess the implementation feasibility and implementation barriers of Teen Well Check. Roll outs of clinical interventions in primary care settings can be a feasible way to address public health concerns.

Acknowledgements

We would like to acknowledge and thank all participants in this research study as well as Dr. Hollie Granato for the initial development of the sexual assault scenarios and Bea Stephens for the comic design.

Disclosure statement

Dr. Levy serves as an expert consultant for the case against JUUL. All other authors have no potential conflicts of interest to report.

Data availability statement

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

Additional information

Funding

Data collection and manuscript preparation was supported by the National Institute on Drug Abuse (K23DA042935; PI: Gilmore). Manuscript preparation also was supported in part by K24DA039783 (PI: Danielson) and the National Institute of Child Health and Human Development (K23HD098325; PI: Ridings).