Developing questionnaires for educational research: AMEE Guide No. 87

Figures & data

Table

Practice points

Questionnaires are widely used in medical education research, yet the processes employed to develop questionnaires vary in quality and lack consistent, rigorous standards. This AMEE Guide introduces a systematic, seven-step design process for creating high-quality survey scales fit for program evaluation and research purposes. The seven-step design process synthesizes multiple techniques survey designers employ into a cohesive process. The survey design process described in this Guide includes the following seven steps: (1) conduct a literature review, (2) carry out interviews and/or focus groups, (3) synthesize the literature review and interviews/focus groups, (4) develop items, (5) collect feedback on the items through an expert validation, (6) employ cognitive interviews to ensure that respondents understand the items as intended and (7) conduct pilot testing. This seven-step design process differs from previously described processes in that it blends input from other experts in the field as well as potential participants. In addition, this process front loads the task of establishing validity by focusing heavily on careful item development.

Table 1. A seven-step, survey scale design process for medical education researchers.

Step	Purpose
1. Conduct a literature review	To ensure that the construct definition aligns with relevant prior research and theory and to identify existing survey scales or items that might be used or adapted
2. Conduct interviews and/or focus groups	To learn how the population of interest conceptualizes and describes the construct of interest
3. Synthesize the literature review and interviews/focus groups	To ensure that the conceptualization of the construct makes theoretical sense to scholars in the field and uses language that the population of interest understands
4. Develop items	To ensure items are clear, understandable and written in accordance with current best practices in survey design
5. Conduct expert validation	To assess how clear and relevant the items are with respect to the construct of interest
6. Conduct cognitive interviews	To ensure that respondents interpret items in the manner that survey designer intends
7. Conduct pilot testing	To check for adequate item variance, reliability and convergent/discriminant validity with respect to other measures

Adapted with permission from Lippincott Williams and Wilkins/Wolters Kluwer Health: Gehlbach et al. (2010). AM last page: Survey development guidance for medical education researchers. Acad Med 85:925.

Table 2. Item-writing “best practices” based on scientific evidence from questionnaire design research.

Pitfall	Survey example(s)	Why it’s a problem	Solution(s)	Survey example(s)	References
Creating a double- barreled item	– How often do you talk to your nurses and administrative staff when you have a problem?	Respondents have trouble answering survey items that contain more than one question (and thus could have more than one answer). In this example, the respondent may talk to his nurses often but talk to administrative staff much less frequently. If this were the case, the respondent would have a difficult time answering the question. Survey items should address one idea at a time.	When you have multiple questions/premises within a given item, either (1) create multiple items for each question that is important or (2) include only the more important question. Be especially wary of conjunctions in your items.	– How often do you talk to your nurses when you have a problem? – How often do you talk to your administrative staff when you have a problem?	Tourangeau et al. 2000; Dillman et al. 2009
Creating a negatively worded item	– In an average week, how many times are you unable to start class on time? – The chief resident should not be responsible for denying admission to patients.	Negatively worded survey items are challenging for respondents to comprehend and answer accurately. Double negatives are particularly problematic and increase measurement error. If a respondent has to say “yes” in order to mean “no” (or “agree” in order to “disagree”), the item is flawed.	Make sure “yes” means yes and “no” means no. This generally means wording items positively.	– In an average week, how many times do you start class on time? – Should the chief resident be responsible for admitting patients?	Dillman et al. 2009
Using statements instead of questions	I am confident I can do well in this course. • Not at all true • A little bit true • Somewhat true • Mostly true • Completely true	A survey represents a conversation between the surveyor and the respondents. To make sense of survey items, respondents rely on “the tacit assumptions that govern the conduct of conversation in everyday life” (Schwarz 1999). Only rarely do people engage in rating statements in their everyday conversations.	Formulate survey items as questions. Questions are more conversational, more straightforward and easier to process mentally. People are more practiced at responding to them.	How confident are you that you can do well in this course?  • Not at all confident • Slightly confident • Moderately confident • Quite confident • Extremely confident	Krosnick 1999; Schwarz 1999; Tourangeau et al. 2000; Dillman et al. 2009
Using agreement response anchors	The high cost of health care is the most important issue in America today.  • Strongly disagree • Disagree • Neutral • Agree • Strongly agree	Agreement response anchors do not emphasize the construct being measured and are prone to acquiescence (i.e. the tendency to endorse any assertion made in an item, regardless of its content). In addition, agreement response options may encourage respondents to think through their responses less thoroughly while completing the survey.	Use construct-specific response anchors that emphasize the construct of interest. Doing so reduces acquiescence and keeps respondents focused on the construct in question; this results in less measurement error.	How important is the issue of high healthcare costs in America today?  • Not at all important • Slightly important • Moderately important • Quite important • Extremely important	Krosnick 1999; Tourangeau et al. 2000; Dillman et al. 2009
Using too few or too many response anchors	How useful was your medical school training in clinical decision making?  • Not at all useful • Somewhat useful • Very useful	The number of response anchors influences the reliability of a set of survey items. Using too few response anchors generally reduces reliability. There is, however, a point of diminishing returns beyond which more response anchors do not enhance reliability.	Use five or more response anchors to achieve stable participant responses. In most cases, using more than seven to nine anchors is unlikely to be meaningful to most respondents and will not improve reliability.	How useful was your medical school training in clinical decision making?  • Not at all useful • Slightly useful • Moderately useful • Quite useful • Extremely useful	Weng 2004

Adapted with permission from Lippincott Williams and Wilkins/Wolters Kluwer Health: Artino et al. 2011. AM last page: Avoiding five common pitfalls in survey design. Acad Med 86:1327.

Figure 1 Visual-design “best practices” based on scientific evidence from questionnaire design research.

Table 3. Examples of various Likert-type response options.

Construct being assessed	Five-point, unipolar response scales	Seven-point, bipolar response scales
Confidence	• Not at all confident • Slightly confident • Moderately confident • Quite confident • Extremely confident	• Completely unconfident • Moderately unconfident • Slightly unconfident • Neither confident nor  unconfident (or neutral) • Slightly confident • Moderately confident • Completely confident
Interest	• Not at all interested • Slightly interested • Moderately interested • Quite interested • Extremely interested	• Very uninterested • Moderately uninterested • Slightly uninterested • Neither interested nor  uninterested (or neutral) • Slightly interested • Moderately interested • Very interested
Effort	• Almost no effort • A little bit of effort • Some effort • Quite a bit of effort • A great deal of effort
Importance	• Not important • Slightly important • Moderately important • Quite important • Essential
Satisfaction	• Not at all satisfied • Slightly satisfied • Moderately satisfied • Quite satisfied • Extremely satisfied	• Completely dissatisfied • Moderately dissatisfied • Slightly dissatisfied • Neither satisfied nor  dissatisfied (or neutral) • Slightly satisfied • Moderately satisfied • Completely satisfied
Frequency	• Almost never • Once in a while • Sometimes • Often • Almost always

Table 4. Examples of commonly used verbal probes.

Type of verbal probe	Example
Comprehension/interpretation	“What does the term ‘continuing medical education’ mean to you?”
Paraphrasing	“Can you restate the question in your own words?”
Confidence judgment	“How sure are you that you have participated in 3 formal educational programs?”
Recall	“How do you remember that you have participated in 3 formal educational programs?” “How did you come up with your answer?”
Specific	“Why do you say that you think it is very important that physicians participant in continuing medical education?”
General	“How did you arrive at that answer?” “Was that easy or hard to answer?” “I noticed that you hesitated. Tell me what you were thinking.” “Tell me more about that.”

Adapted with permission from the Journal of Graduate Medical Education: Willis & Artino 2013. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ 5:353–356.

Gehlbach H, Artino AR, Durning S. 2010. AM last page: Survey development guidance for medical education researchers. Acad Med 85:925

 PubMedGoogle Scholar

Tourangeau R, Rips LJ, Rasinski KA. 2000. The psychology of survey response. New York: Cambridge University Press

 Google Scholar

Dillman D, Smyth J, Christian L. 2009. Internet, mail, and mixed-mode surveys: The tailored design method. 3rd ed. Hoboken, NJ: Wiley

 Google Scholar

Artino AR, Gehlbach H, Durning SJ. 2011. AM last page: Avoiding five common pitfalls of survey design. Acad Med 86:1327

 PubMedGoogle Scholar

Willis GB, Artino AR. 2013. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ 5(3):353–356

 PubMedGoogle Scholar