Impact of survey item wording on reported tobacco use among youth: effect of adding ‘even one or two puffs’ to use questions

ABSTRACT

Having ever used tobacco is a key surveillance metric. Existing tobacco use survey items differ in their inclusion of minimal use language, such as ‘even one or two puffs.’ This study aimed to quantify how minimal use language affects tobacco ever use prevalence estimated from adolescent surveys. Participants (N = 5127) in the 2022 Teens, Nicotine, and Tobacco Project online panel survey of California adolescents (ages 12–17) were randomized to one of two differently worded ever use survey items (i.e. with or without minimal use language) for eight different tobacco products independently. For seven of the eight products (except hookah), minimal use language resulted in numerically higher ever use prevalence estimates. Averaged across all products, ever use prevalence was 0.7-percentage points higher when items included minimal use language (95% CI: 0.1, 1.4). Findings suggest that minimal use language yields modestly higher tobacco use prevalence, with implications for comparing and interpretating surveillance data.

KEYWORDS:

• Adolescent
• surveys and questionnaires, methods
• tobacco products
• survey format effects

Introduction

Having used a tobacco product at least one time, known as tobacco ‘ever use,’ is a critical behavioral development and a key metric in tobacco use surveillance. Among adolescents, smoking just a few puffs is strongly associated with future daily cigarette smoking compared to never puffing at all (Sargent et al., 2017). Yet, while tobacco product use at any level is relevant for tobacco control and research, adolescents who have tried tobacco may not self-identify as tobacco users unless specifically prompted to consider low levels of use (Nichter et al., 1997).

Existing survey items that measure adolescent tobacco use vary in their inclusion of minimal, experimental use. In the United States, the National Youth Tobacco Survey and Population Assessment of Tobacco and Health Study include phrases like ‘even one or two puffs’ or ‘even once or twice’ in items measuring cigarette and e-cigarette ever use, respectively (National Addiction & HIV Data Archive Program, 2024; Office on Smoking and Health, 2022). The National Youth Risk Behavior Survey includes ‘even one or two puffs’ language for cigarettes but not e-cigarettes (Centers for Disease Control and Prevention, 2023). Monitoring the Future survey items do not include such phrasing (Miech et al., 2023). These wording differences pose potential challenges in comparing ever use prevalence across tobacco products and datasets.

The goal of the present study is to quantify how minimal use phrasing, such as ‘even one or two puffs,’ affects the prevalence of tobacco ever use estimated from adolescent surveys. Understanding such effects may inform future survey design and the interpretation of tobacco surveillance data.

Methods

The Teens, Nicotine, and Tobacco (TNT) Project includes an online survey of California adolescents (ages 12–17) recruited using commercial non-probability survey panels, as detailed elsewhere (Chaffee, Couch, Cheng, et al., 2023; Chaffee, Couch, Wilkinson, et al., 2023). Briefly, the 2022 TNT Online Survey included 5,127 participants (total) who completed the survey in English or Spanish in June-July or November-December cycles. Eligibility criteria (age 12–17 and residing in California) were matched to panel-maintained demographic profiles and confirmed in survey responses. Signed parental consent was not collected to maintain anonymity; participants were provided study information before affirming their comprehension and willingness to continue. Participant incentives differed across panels but typically consisted of redeemable reward points valued at $5 or less. The University of California San Francisco Institutional Review Board approved all study procedures (approval number: 21–33292).

Tobacco use survey items were based on standard questions used in national tobacco surveillance and are detailed elsewhere (Chaffee, Couch, Wilkinson, et al., 2023). Each tobacco product (e.g. cigarettes, e-cigarettes, hookah) was assessed using separate items. To aid recognition, for each tobacco product, participants were provided an image and a brief text description that included example brands. Before implementation, TNT survey items underwent qualitative cognitive testing with an independent sample of adolescents (N = 12).

One TNT Project objective is to inform other statewide tobacco surveillance and communication efforts. Thus, an experiment was embedded to determine the effect of including minimal use wording (e.g. ‘even one or two puffs’) on the estimated prevalence of ever tobacco use for individual products (e.g. having ever used a cigarette, having ever used a cigar, etc.). For eight tobacco products, with equal probability and independently by product, participants were randomly assigned to view minimal use wording or control wording (i.e. no minimal use wording) to assess ever use. Assignment was allowed to vary across products within participants (e.g. a participant could view minimal use wording for cigarettes but not for cigars). Items for cigarettes, hookah, big cigars, and little cigars and cigarillos had yes/no response options (i.e. dichotomized use).

Minimal Use Wording (for cigarettes, hookah, big cigars, and little cigars):

Have you EVER smoked [tobacco product], even one or two puffs?

Control Wording (for cigarettes, hookah, big cigars, and little cigars):

Have you EVER smoked [tobacco product]?

Items for e-cigarettes, moist snuff, chewing tobacco, and snus asked about number of times used (i.e. lifetime use), with six response options from never up to 100 or more times.

Minimal Use Wording (for e-cigarettes, moist snuff, chewing tobacco, and snus):

How many times have you used [product] in your entire life, even one or two puffs?*

Response option: I have never [used product], not even [one puff | once].

*‘Puff’ wording used only for e-cigarettes; ‘once or twice’ used for snuff, chew, or snus

Control Wording (for e-cigarettes, moist snuff, chewing tobacco, and snus):

How many times have you used [product] in your entire life?

Response option: I have never [used product].

Prevalence of tobacco product ever use was calculated as the percentage of participants reporting any use at all of a given product (i.e. ‘yes’ for dichotomized use items and any response other than ‘never’ for lifetime use items).

Several steps were taken to improve data quality and generalizability. Percentages were weighted for enhanced generalizability using geodemographic weights developed from American Community Survey 2016–2020 Public Use Microdata Sample files for California (ages 12–17). Excluded were responses with suspicious scaling patterns (e.g. responding the maximum use frequency for all tobacco products) or inappropriate responses in open-text fields. Also excluded were responses that failed a ReCAPTCHA challenge or responded to a ‘hidden’ item (not visible to people but potentially visible to computer bots). Some response attributes were not disqualifying but potentially indicative of lower data quality in online samples, such as completion from midnight to 4:00 am, imperfect ReCAPTCHA scores, or failed attention checks. To reduce the influence of such responses, among all recorded responses (excluded and retained), we fitted a logistic regression model to obtain the predicted probability of being excluded, based on response attributes. Retained responses were weighted by the inverse of that probability. Further details on weights are available elsewhere (Chaffee, Couch, Cheng, et al., 2023).

For each tobacco product, ever use prevalence was compared between minimal use wording and control wording using survey-weighted chi-square tests. To estimate any minimal use wording effect across all products, a generalized estimating equation model was then fitted for the outcome ever use (binomial family, logit link), including minimal use wording as the lone covariable and allowing up to eight observations per participant (exchangeable working correlation structure). Apart from the main model, two separate, additional models were fitted with interaction terms: one for tobacco product and one for item format (dichotomized or lifetime). To express the average magnitude of any minimal use wording effect across all products, the marginal mean absolute difference in ever use prevalence was calculated using the margin command in Stata 16.

Results

The included sample had a mean age of 14.6 years. Female gender (58% versus 38% male and 4% other) and Hispanic/Latino race/ethnicity (52% versus 22% Non-Hispanic White, 14% Asian, and 12% other) were common. Descriptively, ever use of at least one tobacco product was more prevalent among older participants, did not differ meaningfully by gender, was highest among those who identified as Hispanic/Latino, and lowest among those who identified as Asian.

For seven of the eight tobacco products individually, ever use prevalence was higher when survey items included minimal use wording. While the wording effect was only statistically significant for cigarettes, the direction of the effect was the same for all products except hookah.

Table

Cigarettes
Minimal Use Wording: 13.0%	Control Wording: 10.3%	p-value: 0.02
Hookah
Minimal Use Wording: 4.2%	Control Wording: 5.1%	p-value: 0.24
Big Cigars
Minimal Use Wording: 3.1%	Control Wording: 2.3%	p-value: 0.18
Little Cigars
Minimal Use Wording: 6.6%	Control Wording: 5.5%	p-value: 0.21
E-Cigarettes
Minimal Use Wording: 21.4%	Control Wording: 20.5%	p-value: 0.53
Moist Snuff
Minimal Use Wording: 3.2%	Control Wording: 2.7%	p-value: 0.25
Chewing Tobacco
Minimal Use Wording: 2.8%	Control Wording: 2.4%	p-value: 0.30
Snus
Minimal Use Wording: 2.7%	Control Wording: 2.1%	p-value: 0.17

Based on marginal mean differences averaged over all products, minimal use wording was associated with a 0.7-percentage point increase (95% confidence interval: 0.1, 1.4; p = 0.03) in estimated tobacco use prevalence (i.e. 7.1% versus 6.4%). No statistically significant or meaningful interaction was identified, either by product overall (7 d.f.; p = 0.25) or by dichotomized versus lifetime format (p = 0.94).

Discussion

The results of this study suggest that explicitly including minimal use wording, such as ‘even one or two puffs’ leads to modestly higher tobacco use prevalence estimates. While the magnitude of this effect was not statistically significant for most tobacco products individually, it was consistent in direction for all but one product and statistically significant for cigarettes and when averaged over all products.

The minimal use wording effect is likely smaller than some other reported influences of tobacco use prevalence estimates, such as whether an adolescent survey is administered at home or in school (Biglan et al., 2004), or whether products are presented individually or as part of a list (Johnson et al., 2019). However, under the present context of declining levels of tobacco use, a difference approaching one percentage point has potential relevance.

These findings also suggest that adolescents pay attention to survey wording when formulating their responses. This is particularly notable for an online panel survey, a method for which inattentive survey-taking is a potential limitation (Nur et al., 2023; Roulin, 2015). Thus, these results further support the validity of self-report in assessing tobacco use behaviors.

Presumably, the minimal use phrasing prompted participants with only limited experiences trying tobacco to endorse past use while taking the survey. Accurately capturing experimental levels of tobacco use in adolescent surveys is important. For example, smoking just a few puffs of a cigarette is associated with elevated risk of progression to future daily smoking (Sargent et al., 2017). While the present study detected a difference in self-reported tobacco ever use according to survey item wording, it does not necessarily show which format is more accurate. The study lacks biochemical validation, and its cross-sectional design precludes longitudinal evaluation. A question for further research is whether either wording format is more predictive of future behaviors, such as progression to daily tobacco use. Qualitative research could confirm how adolescents conceptualize minimal use phrasing. Additionally, findings should be confirmed in larger, more generalizable, and ideally longitudinal samples.

Conclusions

Survey wording that explicitly encompasses minimal levels of tobacco use appears to have a modest but detectable effect on tobacco ever use prevalence among adolescents. This type of wording should be considered when designing new survey instruments and when comparing prevalence estimates across surveys and over time.

Acknowledgments

Thank you to Monica Wilkinson of the California Department of Public Health and to Nancy Cheng and Niloufar Ameil of the University of California San Francisco for technical assistance. The findings and conclusions in this article are those of the author(s) and do not necessarily represent the views or opinions of the National Institutes of Health, the California Department of Public Health, or the California Health and Human Services Agency.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the California Department of Public Health [CDPH-20-10026]; National Institutes of Health [U54 HL147127].

Notes on contributors

Benjamin W. Chaffee

Benjamin W. Chaffee, is Professor of Oral Epidemiology and Dental Public Health at the University of California San Francisco (UCSF), Director of the UCSF School of Dentistry Program in Global Oral Health, and faculty affiliate of the UCSF Center for Tobacco Control Research and Education. Dr. Chaffee’s research focuses on new and emerging tobacco products (particularly use among adolescents), dental professional engagement in tobacco cessation, and early-life influences of children’s oral health.

Lauren M. Dutra

Lauren M. Dutra, is a social epidemiologist and Research Scientist in Social, Statistical, and Environmental Sciences at RTI International. Dr. Dutra’s research focuses on the impact of policy on tobacco and cannabis outcomes, particularly disparities in these outcomes. At RTI, she directs projects for the Center for Disease Control and Prevention, Food and Drug Administration, and state clients that focus on the evaluation of mass media public health campaigns, youth tobacco surveillance, and setting research agendas for health equity.

Candice D. Donaldson

Candice D. Donaldson, is a Research Scientist for the California Department of Public Health’s Tobacco Prevention Program. Broadly speaking, she investigates how emotions, attitudes, relationships, behaviors, and environmental characteristics impact substance use and misuse, and then applies that knowledge to promote effective health behavior change in vulnerable and underserved children, adolescents, and young adults. Her most recent research efforts are dedicated to supporting social justice and reducing tobacco related disparities in underrepresented youth populations, including racial/ethnic and sexual and/or gender minorities.

Elizabeth T. Couch

Elizabeth T. Couch, is Assistant Adjunct Professor and Project Policy Analyst in the Department of Preventive and Restorative Dental Sciences at the University of California San Francisco (UCSF). Her research interests include tobacco-related behaviors and decision-making among youth and the role of dental professionals in tobacco prevention and cessation.

Omara Farooq

Omara Farooq, is a Research Scientist at the California Department of Public Health’s Tobacco Prevention Program. The majority of her work has focused on improving the health and wellness of underrepresented minority groups in California. Ms. Farooq currently works on a project which focuses on understanding tobacco, nicotine, and cannabis product use behaviors, perceptions, and terminology among California youth.

Xueying Zhang

Xueying Zhang, is chief of the Surveillance Unit in the California Tobacco Prevention Program (CTPP). Ms. Zhang is the subject matter expert for statewide tobacco use surveillance including study design, instrument development, data collection strategies, survey data analysis, and visualization. She oversees statewide surveillance that provides valuable data to inform the intervention and assess the effectiveness of the overall CTPP and its individual components.

Stuart A. Gansky

Stuart A. Gansky, is biostatistician, Professor, and Lee Hysan Chair of Oral Epidemiology in the Division of Oral Epidemiology and Dental Public Health, as well as Associate Dean for Research of School of Dentistry at the University of California San Francisco (UCSF). He directs the UCSF Center to Address Disparities in Children’s Oral Health (known as CAN DO) and co-directs the NIMHD-funded Research Coordinating Center to Reduce Disparities in Multiple Chronic Diseases. Dr. Gansky’s research has focused on health disparities studies, randomized prevention trials, tobacco initiation prevention and cessation research, and methodological issues related to clinical and translational research.