Tackling deceptive responding during eligibility via content-knowledge questionnaires

KEYWORDS:

• Deceptive responding
• eligibility
• clinical studies
• online research

This article is referred to by:

Utilizing content-knowledge questionnaires to assess study eligibility and detect deceptive responding

Deceptive responding from participants has troubled social research for years. This problem becomes particularly conspicuous when the topic under analysis involves relatively private issues, such as sexual behavior or drug use, or when there are incentives, monetary or otherwise, to participate. Illustrating this problem, an intriguing study surveyed 100 participants who had regularly served as volunteers in clinical trials and found that 25% to 33% had either exaggerated or fabricated symptoms to facilitate their participation and, perhaps more worrisome, 75% had concealed information likely to result in their exclusion from the study (1). A recent review claims that overall deception rate among healthy volunteers ranges from 3% to 25% (2). Almost every piece of literature on deceptive responding indicates the need to improve the screening and eligibility techniques to avoid misrepresentation of own’s behavior (3,4). Clinical studies on drug use can employ objective verification (5), yet toxicological measurements can be very expensive and sometimes they are just not feasible. Example of the latter are studies that employ online research (6).

The paper by Strickland and Stoops “Utilizing content-knowledge questionnaires to assess study eligibility and detect deceptive responding” in this issue of the American Journal on Drug and Alcohol Abuse (7) is particularly relevant to improving deceptive responding. This paper puts forward a procedure to improve eligibility when using a crowdsourced sampling. This type of sampling involves the use of online data collection services such as Amazon Mechanical Turk (MTurk), StudyResponse, or Qualtrics Panels. Researchers can upload custom-made research forms to these services and draw participants from a large and, in principle, diverse population (8,9).

Strickland and Stoops describe a novel method for enrollment criteria assessment, particularly useful for online research or situations when objective measurement of drug use is not possible. Specifically, the researchers recruited approximately 4000 volunteers via MTurk and requested that participants complete – in the context of a larger study on decision-making – a screening survey that included self-reporting of cannabis use and a newly designed Cannabis Knowledge Questionnaire (CKQ). Typically, researchers use only the drug use self-report to decide whether or not a given participant is eligible. The CKQ, which represents the major methodological innovation of the study, included a battery of questions on cannabis jargon, paraphernalia associated with its use, and costs and weight of cannabis products, among other questions related to cannabis use culture. The authors then correlated the CKQ responses with the individual’s self-report of cannabis use history. A major strength of the study was the assessment, in a separate sample of participants, of the correspondence of CKQ scores with a urine toxicology screening for cannabis use.

The CKQ showed good internal consistency and, most importantly, the likelihood of positively answering the questions largely and significantly increased as a function of self-reported drug use. Moreover, in the laboratory study, the number of correct responses to the questionnaire was significantly higher in those that tested positive for cannabis in the urine drug test, compared to those that tested negative. It seems, therefore, that the inclusion of knowledge-based questionnaires is a valuable, simple, and economical tool to detect, and therefore reduce, deceptive responding in addiction research. The study adds to prior work (10) indicating that “two-step” procedures are effective to reduce eligibility problems and response inconsistencies. It is important to remark that, to further facilitate the use of CKQ, the authors provide the instrument as supplementary material of the paper, and describe specificity, sensitivity and predictive values for its individual questions and cutoff points.

As with all research, however, the contributions of the paper need to be discussed in the context of limitations. The use of mTurk biases the sample toward college students and people under 50 years old (11), which detracts from the generality of the results. A related issue is the lack of control of the conditions in which the CKQ was answered. The authors acknowledge that participants may have conducted web searches while responding. Perhaps more importantly, the main assumption of the paper is that a good CKQ score will be more likely to be exhibited by someone who is using the substance vs. someone who did not – or rarely – used it. The implication is, however, dependent on the nature of the questions. Questions that ask about drug-related paraphernalia or drug-related culture may be more likely to select eligible participants who are more attuned or connected to the social groups that use the drug. Yet, drug users that are isolated from current trends may be less likely to answer the questions correctly. The authors addressed this problem in the discussion of the paper and suggested that future knowledge-based questionnaires should try to avoid culture-based questions and, instead, focus on questions that inquire on direct substance use (e.g., costs) or substance effects.

Despite these caveats, the study by Strickland and Stoops (7) represents substantial progress toward improving the quality of the eligibility process. In the absence of objective measures, eligibility often relies on a reduced set of questions that are not further corroborated or, after data collection, are slowly and painstakefully combed for errors, inconsistencies, and other signs of response fabrication. The study provides, likely for the first time, evidence that content-knowledge questionnaires are useful to assess study inclusion and exclusion criteria. Ultimately, subsequent work should refine the questionnaire and its implementation.