Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals

References

• American Association of Public Opinion Research. Best practices for survey research. 2018. Available from: https://www.org/Standards-Ethics/Best-Practices.aspx#best6
   Google Scholar
• American Statistical Association. Section on survey research methods. How to conduct pretesting. 1997. Available from: http://www.parkdatabase.org/files/documents/1999_ASA-How-to-conduct-pre-testing_ASA.pdf
   Google Scholar
• Artino AR, Phillips AW, Utrankar A, et al. “The questions shape the answers”: assessing the quality of published survey instruments in health professions education research. Acad Med. 2018;93:456–11. Available from: https://journals.lww.com/academicmedicine/Abstract/publishahead/_The_Questions_Shape_the_Answers____Assessing_the.98067.aspx
   PubMed Web of Science ®Google Scholar
• Artino AR, Rochelle JS, Dezee KJ, et al. Developing questionnaires for educational research: AMEE guide no. 87. Med Teach. 2014;36:463–474.
   PubMed Web of Science ®Google Scholar
• Bennett C, Khangura S, Brehaut J, et al. Reporting guidelines for survey research: an analysis of published guidance and reporting practices. PLoS Med. 2011;8:e1001069. Available from: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001069#s4
   Web of Science ®Google Scholar
• Bradburn N, Sudman S, Wansink B. Questionnaires from start to finish. In Asking questions. The definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires. San Francisco, CA: Jossey-Bass; 2004. p. 317–318.
   Google Scholar
• Colbert CY, Diaz-Guzman E, Myers JD, et al. How to interpret surveys in medical research: a practical approach. Cleve Clin J Med. 2013;80:423–435.
   Web of Science ®Google Scholar
• Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12:229–238.
   PubMed Web of Science ®Google Scholar
• Collins D. Cognitive interviewing: origin, purpose and limitations. In: Collins D, editor. Cognitive interviewing practice. Los Angeles (CA): Sage Publications; 2014. p. 3–27.
   Google Scholar
• Dillman DA, Smyth JD, Christian LM. Aural versus visual design of questions and questionnaires. In: Internet, phone, mail, and mixed mode surveys. The tailored design method. 4th ed. Hoboken (NJ): John Wiley & Sons, Inc.; 2014. p. 169–227.
   Google Scholar
• Fowler FJ. How unclear terms affect survey data. Public Opin Quar. 1992;56:218–231.
   PubMed Web of Science ®Google Scholar
• Groves RM, Fowler FJ, Couper MP, et al. Survey methodology. 2nd ed. Hoboken (NJ): John Wiley and Sons, Inc.; 2009.
   Google Scholar
• Miller K. Introduction. In: Miller K, Willson S, Chepp V, et al., editors. Cognitive interviewing methodology. Wiley series in survey methodology. Hoboken (NJ): John Wiley & Sons, Inc.; 2014. p. 1–5.
   Google Scholar
• Pew Research Center. U.S. survey research. Questionnaire design. Available from: http://pewresearch.org/methodology/u-s-survey-research/questionnaire-design/
   Google Scholar
• Presser S, Couper MP, Lessler JT, et al. Methods for testing and evaluating survey questions. Pub Opin Quarter. 2004;68:109–130.
   Web of Science ®Google Scholar
• Programme for International Student Assessment (PISA). Technical report. The development of the context questionnaires. Paris, France: Organization for Economic Co-operation and Development Publishing; 2009. p. 57. Available from: https://www.oecd.org/pisa/data/42025182.pdf
   Google Scholar
• Rickards G, Magee C, Artino AR. You can’t fix by analysis what you’ve spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ. 2012 Dec;4(4):407–410.
   Google Scholar
• Schwartz N, Oyserman D. Asking questions about behavior: cognition, communication, and questionnaire construction. Am J Eval. 2001;22:127–160.
   Web of Science ®Google Scholar
• Tourangeau R, Rips LJ, Rasinski K. The psychology of survey responses. New York (NY): Cambridge University Press; 2000.
   Google Scholar
• Sullivan GM, Artino AR. How to create a bad survey instrument. J Grad Med Educ. 2017;9:411–415.
   Google Scholar
• Willis GB. Cognitive interviewing training guide. 2012. Available from: www.researchgate.net/gordonbwillis
   Google Scholar
• Willis G. Current developments in the cognitive testing of survey questions. American Association of Public Opion Research (AAPOR) Webinar. 2018 June 1. Available from: https://www.aapor.org/Education-Resources/Online-Education/Webinars-FAQ.aspx
   Google Scholar
• Willis GB, Artino AR. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ. 2013;5:353–356.
   PubMedGoogle Scholar
• Willis G, Royston P, Bercini D. The use of verbal report methods in the development and testing of survey questionnaires. App Cog Psych. 1991;5:251–267.
   Web of Science ®Google Scholar
• Desimone LM, Le Floch KC. Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educ Eval Pol Anal. 2004;26:1–22.
   Web of Science ®Google Scholar
• Duffett M, Burns KE, Adhikari NK, et al. Quality of reporting of surveys in critical care journals: a methodologic review. Crit Care Med. 2012;40:441–449.
   Web of Science ®Google Scholar
• Lavrakas PJ. Encyclopedia of survey research methods. Thousand Oaks (CA): Sage Publications, Inc.; 2008. DOI:10.4135/9781412963947
   Google Scholar
• Billings-Gagliardi S, Barrett SV, Mazor KM. Interpreting course evaluation results: insights from think aloud interviews with medical students. Med Educ. 2004;38:1061–1070.
   PubMed Web of Science ®Google Scholar
• Willson S, Miller K. Data collection. In: Miller K, Willson S, Chepp V, et al., editors. Cognitive interviewing methodology. Wiley series in survey methodology. Hoboken (NJ): John Wiley & Sons, Inc; 2014. p. 15–33.
   Google Scholar
• Krueger RA, Casey MA. Focus groups. A practical guide for applied research. 3rd ed. Thousand Oaks (CA): Sage Publications; 2000.
   Google Scholar
• Diaz-Guzman E, Colbert CY, Mannino DM, et al. 24/7 in-house intensivist coverage and fellowship education: a cross sectional survey of academic medical centers in the USA. Chest. 2012;141:959–966.
   Web of Science ®Google Scholar
• Mullens JE, Kasprzyk D. Using qualitative methods to validate quantitative survey instruments. National Center for Education Statistics. 2002. Available from: https://pdfs.semanticscholar.org/b41a/de60b7918adeffa6d8c7b655b658ff2ab832.pdf?_ga=2.116987523.425779153.1521137932-
   Google Scholar
• Rogers G. Accreditation Board for Engineering and Technology (ABET), Inc. Sample protocol for pilot testing survey items. Available from: http://www.abet.org/wp-content/uploads/2015/04/sample-protocol-pilot-testing-survey-items-pdf
   Google Scholar
• Agency for Healthcare Research and Quality. ESRD toolkit. Using checklists and audit tools. 2015. Available from: https://www.ahrq.gov.resources.esrd
   Google Scholar
• Akkerman S, Admiraal W, Brekelmans M, et al. Auditing quality of research in social sciences. Qual Quant. 2008;42:257–274.
   Web of Science ®Google Scholar
• Copeland GA Clinical governance support team. National health service. A practical handbook for clinical audit guidance. 2005. Available from: http://webarchive.nationalarchives.gov.uk/20081112120728/http://www.cgsupport.nhs.uk/Resources/Clinical_Audit/1@Introduction_and_Contents.asp
   Google Scholar
• Goddard M World health organization. Primary health care continuous quality improvement (CQI) tools. eHealth. Compendium of innovative health technologies for low-resource settings. 2012. Available from: http://www.who.int/ehealth
   Google Scholar
• Organization for Economic Co-Operation and Development. Good practices in supporting supreme audit institutions. 2011. Available from: http://www.oecd.org/dac/effectiveness/Final%20SAI%Good%20Practice%20Note.pdf
   Google Scholar
• Blake M. Other pretesting methods. In: Collins D, editor. Cognitive interviewing practice. Los Angeles (CA): Sage Publishing; 2014. p. 28–56.
   Google Scholar
• Statistical and Science Policy Office, Office of Information and Regulatory Affairs, Office of Management and Budget. Statistical policy working paper 47. Evaluating survey questions: An inventory of methods. 2016 January. Available from: https://nces.ed.gov/FCSM/pdf/spwp47.pdf
   Google Scholar
• Ruark A, Fielding-Miller R. Using qualitative methods to validate and contextualize quantitative findings: a case study of research on sexual behavior and gender-based violence among young Swazi women. Global Health Sci Prac. 2016;4:373–383.
   PubMed Web of Science ®Google Scholar
• Krosnick JA. Survey research. Ann Rev Psych. 1999;50:537–567.
   PubMed Web of Science ®Google Scholar
• Schaeffer NC, Presser S. The science of asking questions. Ann Rev Soc. 2003;29:65–88.
   Web of Science ®Google Scholar
• Willis GB, Miller K. Cross-cultural cognitive interviewing: seeking comparability and enhancing understanding. Field Methods. 2011;23:331–341.
   Web of Science ®Google Scholar
• Martin SL, Birhanu Z, Omotao MO, et al. “I can’t answer what you’re asking me. Let me go, please.”: cognitive interviewing to assess social support measures in Ethiopia and Kenya. Field Methods. 2017;29(4):317–332.
   Web of Science ®Google Scholar
• Podsakoff PM, MacKenzie SB, Podsakoff NP. Sources of method bias in social science research and recommendations on how to control it. Ann Rev Psych. 2012;63:539–569.
   PubMed Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. Validity. In: Standards for educational and psychological testing. Washington (DC): American Educational Research Association; 2014. p. 11–31.
   Google Scholar
• Schwarz N. Self-reports. How the questions shape the answers. Am Psych. 1999;54:93–105. Available from: http://psycnet.apa.org/fulltext/1999-00297-001.pdf
   Web of Science ®Google Scholar
• National Center for Education Statistics. Planning and design of surveys. Available fom: https://nces.ed.gov/statprog/2002/std2_1.asp
   Google Scholar
• Ho-Ting Li A, Thomas SM, Farag A, et al. Quality of survey reporting in nephrology journals: a methodologic review. Clin J Am Soc Neph. 2014;9:2089–2094.
   Web of Science ®Google Scholar
• Minooee S, Masoumeh S, Sheikhan Z, et al. Audit of intrapartum care based on national guidelines for midwifery and birth services. Eval Health Prof. 2018;1–15. DOI:10.1177/0163278718778095
   Web of Science ®Google Scholar
• Phitayakorn R, Macklin EA, Goldsmith J, et al. Applicants’ self-reported priorities in selecting a residency program. J Grad Med Educ. 2015;7:21–26.
   PubMedGoogle Scholar
• Cook AF, Arora VM, Rasinski KA, et al. The prevalence of medical student mistreatment and its association with burnout. Acad Med. 2014;89: 759–754.
   Web of Science ®Google Scholar
• Sutton PA, Mason J, Vimalachandran D, et al. Attitudes, motivators, and barriers to a career in surgery: a national study of UK undergraduate medical students. J Surg Educ. 2014;71:662–667.
   Web of Science ®Google Scholar
• Pollart SM, Dandar V, Brubaker L, et al. Characteristics, satisfaction, and engagement of part-time faculty at U.S. medical schools. Acad Med. 2015;90:355–364.
   Web of Science ®Google Scholar
• DeCastro R, Griffith KA, Ubel PA, et al. Mentoring and the career satisfaction of male and female academic medical faculty. Acad Med. 2014;89:302–311.
   Web of Science ®Google Scholar
• Fonseca AL, Reddy V, Longo WE, et al. Are graduating surgical residents confident in performing open vascular surgery? Results of a national survey. J Surg Educ. 2015;72:577–584.
   Web of Science ®Google Scholar
• Stevens J, Johansson A, Lennes I, et al. Long-term culture change related to rapid response system implementation. Med Educ. 2014;48:1211–1219.
   Web of Science ®Google Scholar
• Kirkpatrick JD, Kirkpatrick WK. Four levels of training evaluation. Alexandria (VA): ATD Press; 2016.
   Google Scholar
• Derzon JH. Challenges, opportunities and methods for large-scale evaluations. Eval Health Prof. 2018;41:321–345.
   Web of Science ®Google Scholar
• Pedhazur EJ, Pedhazur Schmelkin L. Criterion-related validation. In: Measurement, design and analysis: an integrated approach. Hillsdale (NJ): Lawrence Erlbaum and Associates; 1991. p. 30–32.
   Google Scholar
• Burns KE, Duffett M, Kho ME, et al., ACCADEMY Group. A guide for the design and conduct of self-administered surveys of clinicians. Can Med Assoc J. 2008;179:245–252.
   PubMed Web of Science ®Google Scholar
• Downing SM. Threats to the validity of clinical teaching assessments: what about rater error? Med Educ. 2005;39:350–355.
   PubMed Web of Science ®Google Scholar
• Group on Educational Affairs, Medical Education Scholarship Research and Evaluation Section. Annotated bibliography of journals for educational scholarship, revised. 2017. Available from: https://www.aamc.org/download/456646/data/annotatedbibliography-of-journals-july-2017.pdf
   Google Scholar
• Chepp V, Gray C. Foundations and new directions. In: Miller K, Willson S, Chepp V, et al., editors. Cognitive interviewing methodology. Wiley series in survey methodology. Hoboken (NJ): John Wiley & Sons, Inc. 2014. Chapter 2, p. 1–5.
   Google Scholar
• Benitez Baena I, Padilla JL. Cognitive interviewing in mixed research. In: Miller K, Willson S, Chepp V, et al., editors. Cognitive interviewing methodology. Wiley series in survey methodology. Hoboken (NJ): John Wiley & Sons, Inc. 2014. Chapter 9, p. 152.
   Google Scholar
• Willis G, Boeije H. The survey field needs a framework for the systematic reporting of questionnaire development and pretesting. Methodology. 2013;93:85–86.
   Google Scholar