Quality and Impact of Survey Research Among Anesthesiologists: A Systematic Review

Abstract

New technology has facilitated survey research of anesthesia professional society members. We evaluated prevailing metrics of quality and impact of published research studies based on surveys of anesthesiologists. We hypothesized that adherence to recommended practices (such as use of reminders) would be associated with increased survey response rates, and that higher response rates would be associated with higher article impact. Using the MEDLINE database, we identified 45 English-language research articles published in 2010–2017 reporting original data from surveys of anesthesiologists. The median response rate was 37% (IQR: 25–46%). Recommended survey practices, including the use of reminders (p = 0.861) and validated questionnaires (p = 0.719), were not correlated with response rates. In turn, survey response rates were not associated with measures of article impact (p = 0.528). The impact of published research based on surveys of anesthesiologists, as measured by citation scores (p = 0.493) and Altmetrics (p = 0.826), may be driven primarily by the novel data or questions raised using survey methodology, but does not appear to be associated with response rates. Improving reporting of survey methodology and understanding possible sources of non-response bias are important for future studies in this area.

Keywords:

• survey methodology
• anesthesiologist
• response rate
• survey research
• systematic review

Introduction

Survey research has been used by investigators for studying clinical, educational, and professional topics in the field of anesthesiology. The feasibility of such research has been facilitated by the prevalence of computer technology, the increased ease of communication via the internet, the development of online survey questionnaires, and the availability of e-mail lists from various organizations for participant recruitment. However, the validity of survey data is reduced by low response rates, missing data points, and poorly designed questionnaires.¹ The importance of appropriate survey design for enhancing inference from survey data is increasingly recognized among clinical researchers.^2,3 Recent guidelines for survey research on clinical topics have emphasized the need for reducing potential sources of bias by using validated questions, pre-testing survey questionnaires, and using incentives or reminders to enhance response rates.^4–6 However, the value of adopting these “best practices” of survey research has not been examined for studies recruiting the participation of anesthesiologists. Understanding the quality and impact of data that can be generated from surveys of anesthesiologists can support development of higher-quality surveys among members of the profession and increase the value of members’ participation in surveys.

In this systematic review, we identified published survey research sampling anesthesiologists and evaluated the quality and impact of these studies. The primary measure of quality was the survey response rate.⁶ Measures of impact included article citations, publication in high impact factor journals, Altmetric scores, and mention of studies in educational materials, news media, and professional society publications (other than academic journals). We hypothesized that adherence to recommended practices of survey design was associated with higher survey response rate and higher article impact. Our secondary aim was to determine which survey characteristics were associated with improved survey response rate, or greater impact, of surveys conducted among anesthesiologists.

Methods

IRB approval was not necessary for this study because it was a review of published research articles. In April and May of 2018, we conducted a systematic search of the MEDLINE database for English-language peer-reviewed research articles published in 2010–2017, which reported original data generated by surveying anesthesiologists based on their membership in one of the following professional societies: American Society of Anesthesiologists, ASA; Association of Anaesthetists of Great Britain and Ireland, AAGBI; Canadian Anesthesiologists’ Society, CAS; Australian and New Zealand College of Anaesthetists, ANZCA; Society for Pediatric Anesthesia, SPA; Association of Paediatric Anaesthetists of Great Britain and Ireland, APAGBI; Canadian Pediatric Anesthesia Society, CPAS; and Society for Pediatric Anaesthesia in New Zealand and Australia, SPANZA. Search keywords included “survey” and “anesthesia” or “anesthesiologist”/“anesthetist” (using British and American spelling, as well as plural and singular nouns). We used PubMed to search for articles, filtering results according to journal indexing in MEDLINE.

We identified titles and abstracts of articles and evaluated each for potential inclusion in the systematic review. We obtained full texts for original research articles (full length or brief communication formats) meeting our inclusion criteria. We screened each paper for original survey research of individual anesthesiologists responding on their own behalf. We excluded studies that sampled institutions or asked respondents to participate in their study on behalf of an institution (eg, studies that targeted fellowship program directors), and qualitative research using entirely unstructured surveys or interviews. We resolved disagreements regarding whether an article met inclusion criteria through discussion among the investigators. The lead investigator reviewed articles selected for inclusion, and coded the pre-specified fields shown in Table 1. All studies meeting inclusion criteria were planned to be included in the analysis, and no a priori power calculation was performed. Survey characteristics and quality outcomes which we expected would be commonly reported, and which would be suitable for objective assessment, were pre-selected to be included in the review. Data that were deemed unlikely to be reported in published studies, such as questionnaire visual design, were not included.

Table 1 Pre-Specified Fields Coded for Each Manuscript

	Variable and Categories
Survey characteristics	Age focus of primary society: General, Pediatric
	Primary mode: Web, Paper
	Whether survey questions were previously validated
	Whether pre-test of survey questionnaire was performed
	Whether pre-notification of eligible respondents was used
	Whether incentives were used
	Whether reminders for selected respondents were used
	The number of reminders used
	Duration of data collection in months
	Year data collection was completed
	Primary topic: Anesthetic practice, Other clinical practice, Medical education, Population health, Business or professional topic, Other topics
Quality Outcomes	Survey response rate
	Item nonresponse rate for primary outcome
	Fraction of missing information
	Whether missing data were imputed for analysis
Impact Outcomes	Article citations in Clarivate Analytics Web of Science
	Article citations in Google Scholar
	Altmetric score
	Journal Impact Factor
	Any references in official society publication
	Any references in news media
	Any references in textbooks or edited volumes

The primary quality outcome in this review was the reported survey response rate, defined as the number of completed responses out of the number of potential respondents who were invited to participate. Partially completed surveys were included in the response rate if they were described as part of the number of completed responses in the original studies. Secondary quality measures included the item non-response rate for the primary study outcome, if one was specified, and the fraction of missing information. The item non-response rate was calculated as the number of responses missing data on the primary study outcome out of the total number of survey responses. The fraction of missing information was defined as the number of surveys with any incomplete data on study variables, out of the total number of complete surveys. We also noted whether the studies used multiple imputations to complete any items missing data for analysis. Article impact outcomes were assessed at the time of the review and included article citation counts from Web of Science and Google Scholar; the Altmetric score of news media and social media mentions; and the impact factor of the journal in which the article was published. These characteristics have been used in prior bibliometric research assessing the impact of publications in medicine and life sciences.^7–9 Additionally, we used a Web search (Google) to determine whether articles were mentioned in news media, textbooks or edited volumes, or official publications of professional societies, excluding academic journals.

Characteristics of survey design included the use of validated questions, survey pre-testing, use of incentives for survey completion, pre-notification of invited respondents, and use of reminders, reflecting general recommendations for improving survey research on clinically relevant topics.^4–6 Additional characteristics of surveys that may have influenced the response rate, such as survey mode and duration of data collection, were coded as summarized in Table 1.¹⁰ The study was not powered for a specific primary hypothesis test but explored a range of plausible associations among survey characteristics, survey response rate, and study impact. Response rates and impact measures were compared against survey characteristics using Spearman correlation coefficients and rank-sum tests. We did not assess the risk of bias because the surveys described included studies that evaluated a diverse range of topics, so bias in the estimation of a specific quantity or association was not an area of focus for our review. Data analysis was performed using Stata/IC 14.2 (College Station, TX: StataCorp LP) and two-tailed P<0.05 was considered statistically significant.

Results

Our initial MEDLINE search identified 1448 publications which were reviewed for potential inclusion in the study. Based on title and abstract review, 1403 publications were excluded, and full texts were obtained for the remaining 45 publications. Bibliographic data and study characteristics for the included studies are summarized in Table 2. Twenty surveys were primarily conducted in the United States (US), compared to 16 in Australia/New Zealand, 5 in Canada, and 4 in Great Britain and Ireland. Sample sizes ranged from 84 to 8178, for a total of 35,177 responses among the articles that reported sample size. Most surveys elicited respondents’ opinion about various anesthetic practices, such as the use of laryngeal mask airways, the prevalence of general anesthesia without intravenous access, perioperative management of patients with obstructive sleep apnea, and use of a difficult airway cart.

Table 2 Bibliographic Data and Study Characteristics for the Included Studies

Article Reference	Societies Surveyed	Primary Country	Primary Age of Focus	Survey Year	Number of Responses
Downey et al Anaesth Intensive Care 2017;45:73–78^Citation31	ANZCA	Australia, New Zealand	General	^a	427
Keon-Cohen et al Anaesth Intensive Care 2017;45:396–402^Citation32	ANZCA	Australia, New Zealand	General	^a	290
McCawley et al Anaesth Intensive Care 2017;45:624–630^Citation33	ANZCA	Australia, New Zealand	General	^a	295
Toledo et al Anaesth Analg 2017;123:1611–1616^Citation34	ASA	United States	General	2015	299
Ard et al A&A Case Rep 2016;6:208–16^Citation35	ASA	United States	General	2013	2189
Cordovani et al Can J Anaesth 2016;63:16–23^Citation36	CAS	Canada	General	2012	458
Gurunathan et al Anaesth Intensive Care 2016;44:111–8^Citation37	ANZCA	Australia, New Zealand	General	2014	245
Heard et al Anesth Analg 2016;122:1614–24^Citation38	ANZCA	Australia, New Zealand	General	^a	755
Leslie et al Anaesth Intensive Care 2016;44;291–7^Citation39	ANZCA	Australia, New Zealand	General	2015	395
Rosen et al Paediatr Anaesth 2016;26:207–12^Citation40	CPAS	Canada	Pediatric	2013	106
Sathyamoorthy et al J Clin Anesth 2016;22:266–72^Citation41	SPA	United States	Pediatric	2014	805
Baird et al Anesthesiology 2015;123:997–1012^Citation21	ASA	United States	General	2013	8178
Corcoran et al Anaesth Intensive Care 2015;43:167–74^Citation42	ANZCA	Australia, New Zealand	General	^a	333
Fernandez et al Anesth Analg 2015;120:837–43^Citation43	ASA	United States	General	^a	609
Patel et al Paediatr Anaesth 2015;25:1127–31^Citation44	SPA	United States	Pediatric	2012	743
Raphael et al Anesth Analg 2015;121:1244–99^Citation45	ASA	United States	General	2014	871
Ben-Menachem et al Anesth Analg 2014;119:1180–5^Citation46	ANZCA	Australia, New Zealand	General	2012	289
Cote et al Anesth Analg 2014;118:1276–83^Citation47	SPA	United States	Pediatric	^a	731
De Oliveira et al Anesth Analg 2014;120:209–13^Citation48	ASA	United States	General	^a	641
Lavi et al Can J Cardiol 2014;30:627–33^Citation49	CAS	Canada	General	2013	497
Schroeck et al Int J Pediatr Otorhinolaryngol 2014;78:2140–4^Citation50	SPA	United States	Pediatric	^a	322
Wong et al Can J Anaesth 2014;61:717–26^Citation51	CAS	Canada	General	2013	997
Afonso et al J Clin Anesth 2013;25:289–95^Citation52	ASA	United States	General	2009	304
Bradley et al Paediatr Anaesth 2013;23:1006–9^Citation53	APAGBI	United Kingdom	Pediatric	^a	^a
Fahy et al Anaesth Intensive Care 2013; 41:102–7^Citation54	SPANZA	Australia, New Zealand	Pediatric	2009	84
Hall et al Can J Anaesth 2013;60:1170^Citation55	CAS	Canada	General	2012	1293
McDonnell et al Anaesth Intensive Care 2013;41:641–7^Citation56	ANZCA	Australia, New Zealand	General	^a	191
Phillips et al Anaesth Intensive Care 2013;41:374–9^Citation57	ANZCA	Australia, New Zealand	General	2011	678
Raghunathan et al Anesth Analg. 2013;116:644–8^Citation58	ASA	United States	General	2010	1300
Calder et al Paediatr Anaesth 2012;22:1150–4^Citation59	APAGBI, CPAS, SPANZA	United Kingdom	Pediatric	2011	693
Gazoni et al Anesth Analg 2012;114:596–603^Citation60	ASA	United States	General	2009	659
Heard et al Anesth Analg 2012;114:604–14^Citation61	ANZCA	Australia, New Zealand	General	^a	433
McCunn et al J Clin Anesth 2012;24:38–43^Citation62	ASA	United States	General	2010	460
McGain et al Anesth Analg 2012;114:1049–54^Citation63	ANZCA	Australia, New Zealand	General	2009	210
Orkin et al Anesthesiology 2012;117:953–63^Citation64	ASA	United States	General	^a	3222
Pettigrew et al Paediatr Anaesth 2012;22:438–41^Citation65	AAGBI, APAGBI	United Kingdom	Pediatric	2010	727
Vigoda et al J Clin Anesth 2012;24:446–55^Citation66	ASA	United States	General	^a	1595
Cannesson et al Crit Care 2011;15:R197^Citation67	ASA	United States	General	^a	210
Firth et al Paediatr Anaesth 2011;21:43–9^Citation68	SPA	United States	Pediatric	2009	510
Trentman et al J Clin Comput 2011;25:129–35^Citation69	ASA	United States	General	2010	615
Braun et al Anaesth Intensive Care 2010;38:935–8^Citation70	ANZCA	Australia, New Zealand	General	^a	146
Dooney et al Anaesth Intensive Care 2010;38:354–8^Citation71	ANZCA	Australia, New Zealand	General	2007	306
Homer et al Paediatr Anaesth 2010;20:638–46^Citation72	AAGBI	United Kingdom	Pediatric	2008	310
Nelson et al Anesth Analg 2010;110:754–60^Citation73	SPA	United States	Pediatric	^a	294
Zugai et al Anaesth Intensive Care 2010;38:27–32^Citation74	ANZCA	Australia, New Zealand	General	^a	250

Note: ^aNot reported in article.

Abbreviations: AAGBI, Association of Anesthetists of Great Britain and Ireland; APAGBI, Association of Paediatric Anaesthetists of Great Britain and Ireland; CPAS, Canadian Pediatric Anesthesia Society; CAS, Canadian Anesthesiologists’ Society; SPA, Society for Pediatric Anesthesia; ASA, American Society of Anesthesiologists; ANZCA, Australian and New Zealand College of Anaesthetists.

Survey methodologies for included studies are summarized in Table 3. Most surveys were completed online, although five surveys used paper questionnaires. Reported methods to ensure survey validity included using validated questions (12/45) and pre-testing the survey questionnaire (23/45). Only one survey reported using both a pre-notification and incentives to increase participation, while 4 surveys reported using incentives alone. By contrast, most surveys used one or more reminders to increase participation (33/45, using a median of 2 reminders). Survey response rates, summarized in Table 4, ranged from 7% to 95% (median [IQR]: 37% [25%, 46%]), although the highest response rate attained on an online survey was 67%. Two studies reported the nonresponse rate for the primary outcome (1.4% and 19.1%, respectively), and 12 studies reported an overall fraction of missing information, ranging from 1.3% to 9.3% (median: 5%; IQR: 2%, 6%).

Table 3 Reported Characteristics of Survey Methodologies for Selected Studies

Article Reference	Survey Mode	Survey Duration (months)	Used Validated Questions	Pre-Tested Questionnaire	Pre-notified Respondents	Used Incentives	Number of Reminders
Downey et al^Citation31	Web		Yes				0
Keon-Cohen et al^Citation32	Web	2		Yes			2
McCawley et al^Citation33	Web	2					0
Toledo et al^Citation34	Web		Yes				3
Ard et al^Citation35	Web	4					1
Cordovani et al^Citation36	Web	2		Yes			0
Gurunathan et al^Citation37	Web	2		Yes			1
Heard et al.^Citation38	Paper			Yes			3
Leslie et al^Citation39	Web	1					1
Rosen et al^Citation40	Web	2		Yes			2
Sathyamoorthy et al^Citation41	Web	3					2
Baird et al^Citation21	Web	2	Yes				4
Corcoran et al^Citation42	Web			Yes		Yes	1
Fernandez et al^Citation43	Web			Yes		Yes	2
Patel et al^Citation44	Web	2	Yes				2
Raphael et al^Citation45	Web	3	Yes	Yes			0
Ben-Menachem et al^Citation46	Web		Yes	Yes			1
Cote et al^Citation47	Web	4					2
De Oliveira et al^Citation48	Web						1
Lavi et al^Citation49	Web	9		Yes		Yes	1
Schroeck et al^Citation50	Web		Yes	Yes			1
Wong et al^Citation51	Web	2	Yes				2
Afonso et al^Citation52	Paper	1	Yes	Yes			0
Bradley et al^Citation53	Web						0
Fahy et al^Citation54	Web	6					0
Hall et al^Citation55	Web	6		Yes			3
McDonnell et al^Citation56	Web		Yes	Yes			0
Phillips et al^Citation57	Web	2		Yes			2
Raghunathan et al^Citation58	Web	2	Yes				0
Calder et al^Citation59	Web	4					0
Gazoni et al^Citation60	Paper	3		Yes	Yes	Yes	2
Heard et al^Citation61	Paper			Yes			0
McCunn et al^Citation62	Web	1		Yes			2
McGain et al^Citation63	Web			Yes			2
Orkin et al^Citation64	Web			Yes	Yes		0
Pettigrew et al^Citation65	Web	1					0
Vigoda et al^Citation66	Web		Yes	Yes			1
Cannesson et al^Citation67	Web						2
Firth et al^Citation68	Web	3					2
Trentman et al^Citation69	Web	1					2
Braun et al^Citation70	Web			Yes			1
Dooney et al^Citation71	Web	2		Yes			0
Homer et al^Citation72	Web	3					2
Nelson et al^Citation73	Web						0
Zugai et al^Citation74	Paper					Yes	1

Table 4 Survey Quality and Impact Metrics

Article Reference	Response Rate	Primary Outcome Non-Response	Fraction of Missing Information	Article Citations (Web of Science)	Article Citations (Google Scholar)	Altmetric Score	Impact Factor	Where Article Referenced
Downey et al^Citation31	42.8%	1.4%	0.2%	4	2		1.7	Official society publication
Keon-Cohen et al^Citation32	38%	5.2%		1	1		1.7
McCawley et al^Citation33	29.8%			0	0		1.7
Toledo et al^Citation34	54%			0	2		4.0	Official society publication
Ard et al^Citation35	42%		6.9%	2	4			Textbook
Cordovani et al^Citation36	26%			4	11		2.3	Textbook
Gurunathan et al^Citation37	24.6%			3	3		1.7
Heard et al^Citation38	48.9%			0	1		4.0
Leslie et al^Citation39	41%	3.4%		2	4		1.4
Rosen et al^Citation40	51%			3	5	3	2.3
Sathyamoorthy et al^Citation41	28%			0	3		1.7	Official society publication
Baird et al^Citation21	25.6%			8	7	105	5.2	Official society publication, news media
Corcoran et al^Citation42	33%			5	6		1.7
Fernandez et al^Citation43	18.2%			13	20		4.0	Official society publication
Patel et al^Citation44	27.1%			8	11	1	1.8
Raphael et al^Citation45	14.5%	1.4%		4	8	2	4.0	Official society publication
Ben-Menachem et al^Citation46	30%			5	8		4.0	Textbook
Cote et al^Citation47	30%		4.9%	47	85	5	4.0	Textbook
De Oliveira et al^Citation48	42.7%			17	18		4.0	News media
Lavi et al^Citation49	12.7%			5	12		4.4
Schroeck et al^Citation50	11%			2	5		1.2	Textbook
Wong et al^Citation51	39%		4.2%	15	27		2.3	Textbook
Afonso et al^Citation52	95%			2	2		1.7
Bradley et al^Citation53				11	20	1	2.3
Fahy et al^Citation54	41.6%			2	1		1.7	Textbook
Hall et al^Citation55	67%		8.2%	14	22		2.5	News media, textbook
McDonnell et al^Citation56	38%			4	7		1.7	Official society publication, news media
Phillips et al^Citation57	38.9%			15	20		1.7	News media
Raghunathan et al^Citation58	13.5%	19.1%		2	6	0	4.0	Textbook
Calder et al^Citation59	28.8%			6	12	1	2.3	Official society publication
Gazoni et al^Citation60	56%			30	60	6	4.0	Textbook
Heard et al^Citation61	49%		4.8%	19	46	5	4.0
McCunn et al^Citation62	8.1%		9.3%	9	17		1.7	Official society publication
McGain et al^Citation63	41%			5	18		4.0	Official society publication, news media
Orkin et al^Citation64	36.2%			15	26	14	5.8	News media, textbook
Pettigrew et al^Citation65	51.2%			2	3	1	1.8	Official society publication
Vigoda et al^Citation66	7.1%		2.2%	0	13		1.7	Textbook
Cannesson et al^Citation67	8.4%			135	218	12	5.4	Official society publication, news media, textbook
Firth et al^Citation68	25%			4	10	1	2.3
Trentman et al^Citation69	12.3%			24	35		2.2
Braun et al^Citation70	29%			0	16		1.7	Official society publication, news media
Dooney et al^Citation71	52.8%		1.3%	5	4		1.7	Textbook
Homer et al^Citation72	52%			8	10	1	2.3
Nelson et al^Citation73	42%			29	47	2	4.0	Textbook
Zugai et al^Citation74	55%			0	12		1.7

Considering the use of reminders, incentives, pre-notification, questionnaire pre-testing, and use of validated questions, we identified 30 studies which used at least one of these methods, and 15 studies which used none of these methods. The survey response rates did not significantly differ between these two groups (median: 37% vs 36%, p = 0.544). Considering other survey characteristics, median response rates were lower in US surveys compared to non-US surveys (26% vs 39%; 95% confidence interval [CI] of difference: 2%, 23%; p = 0.021), and higher for paper surveys compared to web surveys (55% vs 30%; 95% CI of difference: 13%, 42%; p=0.002). There were not enough data to compare secondary quality outcomes, such as the fraction of missing information.

Other study impact metrics include article citation counts, which ranged from 0 to 218 in Google Scholar (median [IQR]: 10 [4, 20]), and from 0 to 135 in Web of Science (median [IQR]: 5 [2, 14]). Among 16 articles from publishers reporting Altmetric scores, these scores ranged from 0 to 105 (median [IQR]: 2 [1, 6]). All except one of the surveys were published in journals with an assigned 2016 impact factor, with scores ranging from 1.2 to 5.8. Thirteen of the surveys have been referenced in official society publications, 15 in textbooks, and nine in news media. The survey response rate was not correlated with article citations on Google Scholar (ρ = −0.17, p = 0.251) citations on Web of Science (ρ = −0.12, p = 0.450), Altmetric scores (ρ = 0.06, p = 0.826) or journal impact factor (ρ = −0.005, p = 0.976). Survey response rates did not differ between studies that were referenced in official society publications, textbooks, or news media (median [IQR]: 36% [26%, 42%]) and studies that were not (median [IQR]: 38% [25%, 51%]; p = 0.544).

Discussion

Increased feasibility of administering surveys has prompted many groups to conduct survey research of anesthesiology professional societies, in order to gather novel data on clinical practices and to explore professional issues in the field. While recent overviews have brought attention to important aspects of survey design, empirical data remain scarce on what defines high-quality research involving surveys of anesthesiologists. To address this, we reviewed published surveys of large English-language professional anesthesia societies, focusing on variation and correlation in measures of survey quality and impact. Our review identified limited reporting of survey characteristics and a wide variability in survey response rates (7–95%). Other than the use of paper surveys, there were no evident associations between elements of survey design and survey response rates.

Response rate is the primary metric used to assess the quality of survey research.^11,12 Specifically, 23 of the articles reviewed (reporting response rates of 8–55%) cited a low response rate as one of their study limitations. Current research suggests that a response rate of 50–60% could minimize the risk of non-response bias, although 35 of the 45 surveys that were published and were included in our study did not meet this threshold.¹³ The low response rates in many of the surveys reviewed may be due to lower response rates seen in online surveys as compared to paper surveys.^14,15 Nevertheless, the median response rate among surveys included in our review was similar to that found in Sheehan’s meta-analysis, which indicated a median response rate of 37% for web-based surveys.¹⁶ Efforts to increase response rates may include financial incentives, advance letters, attempts to convert respondents who refuse to participate, and follow-up reminders. However, while these techniques have produced higher response rates in some experimental settings, their use has not been definitively associated with a reduction in non-response bias.¹⁷ Thus, high response rates to a survey may be necessary, but not sufficient to assure survey validity.¹⁸ Although our study did not overtly measure non-response bias, it was notable that no survey characteristics other than survey mode were correlated with the response rate. Given the generally high and variable non-response rates in the studies reviewed, we speculate that non-response in this setting could often be caused by anesthesiologists choosing to participate in a given survey primarily based on the topic of the research and its relevance to their practice and interests.

Our review presents new data on the state of survey research in anesthesiology but is subject to some limitations. First, we have focused on surveys of anesthesiology professional society members, which are facilitated by the availability of society mailing lists, and the probable interest of society members in contributing to academic research. Therefore, our conclusions may not be generalizable to surveys of other populations, such as patients or caregivers. We also excluded surveys that sample institutions (eg, surveys of fellowship program directors about characteristics of their fellowship program), as response rates to these surveys tend to be very high, possibly owing to respondents’ perceived responsibility to complete the survey on behalf of their institution.^19,20 Furthermore, we evaluated quality and impact metrics among published studies, but did not analyze which survey characteristics influenced the likelihood of publication and did not include surveys reported only in the “grey literature” (eg, non-peer-reviewed reports) or surveys conducted internally within professional societies. An additional limitation of examining impact metrics is that only a small number of articles receive scholarly and public attention more than the typical article, such as the Baird et al paper.²¹ Our review was further limited to publications in MEDLINE-indexed journals. Among the societies included in the study, all affiliated journals (eg, Anaesthesia, Anesthesiology, Canadian Journal of Anaesthesia, and Anaesthesia and Intensive Care) are indexed in MEDLINE, and some studies have described using MEDLINE as a “white list” of high-quality medical journals.²² Therefore, our review did not address the possibility that some surveys with methodological issues may have been published in journals not indexed in MEDLINE, and may not have captured articles that were published ahead of print during the review period, but not indexed in PubMed, as of May 2018. Lastly, we observed no statistically significant associations and weak correlations for a wide range of plausible of associations among survey characteristics, survey response rate, and study impact. This supports our conclusions regarding the unpredictability of response rates, and an evident lack of association between survey response rates and study impact.

In this study, we utilized bibliometric analyses to quantify the attention scientific articles receive.^7–9,23 One of the better-known aspects of bibliometrics is citation analysis.²⁴ Our review included several highly cited articles, yet we found that the survey response rate was not correlated with the number of citations. Thus, while survey response rates may influence journals’ decision to publish an article, this study characteristic does not appear to influence academic audiences’ subsequent judgment of the study’s importance. Rather, Falagas et al determined that characteristics such as article length and journal impact factor influence citation counts.²⁵ While citation counts are a classic tool of bibliometrics, alternative metrics are gaining popularity for assessing the impact of research outside of academia. The Altmetric score of a research article indicates the amount of online attention it has received by combining information from multiple data sources, such as social media and news mentions.²⁴ In this review, one article had a very high Altmetric score while the rest of the studies received little attention according to this metric. As with citation counts, the response rate was not correlated with the Altmetric scores. This further suggests that the value of survey research in anesthesiology professional societies may be to raise novel questions or present data unobtainable in other ways, but not necessarily to generate precise and unbiased estimates dependent on a high response rate.

In previous studies, aspects of survey methodology were often reported inconsistently and only partially.²⁶ Reviewing articles in anesthesiology journals, Story et al found that the reporting of survey methods was inconsistent and potentially compromised the transparency and reproducibility of surveys.²⁷ Many sources, including but not limited to review articles published in the anesthesiology literature, outline good practices in conducting and reporting survey research, such as robust testing and development of the research instrument.^1–6 Kelley et al emphasize the importance of reporting the details of primary data collection, such as how participants were selected, as well as how data were analyzed and whether any adjustments were made to account for nonresponse, missing data, or differential probability of response.²⁸ Additionally, Davern et al have discussed the importance of estimating nonresponse bias, such as by comparing the survey with other sources, analyzing para-data, or analyzing external data sources.¹² Halbesleben et al discussed specific formulas to calculate nonresponse bias based on the response rate, proportion of non-respondents, and characteristics of non-respondents.²⁹ Despite these recommendations, one study in this review did not report the number of participants, while only 12 studies reported using previously validated questions and 23 studies reported pre-testing surveys. Only three studies used statistical weights to account for differential probability of response, and only two studies reported primary outcome nonresponse rates.

Techniques such as weighting and imputation, used in survey data analysis to address potential bias and non-response, appear to be under-used in surveys of anesthesia professional societies. Based on this review, several recommendations for survey practice can be made. Investigators should report essential information regarding data collection and analysis, such as the population targeted, the sampling frame, and whether the sampling frame included the entire society, a random subsample, or a non-random subsample (eg, attendees at a conference). Studies should also describe the development of survey research tools, the source of survey questions, the response rate, the primary outcome nonresponse rate, and the fraction of missing information, using established definitions from the survey methodology literature as appropriate. Additionally, researchers should attempt to analyze nonresponse bias, which can be significant even in the presence of high response rates.^12,29 It is particularly important to consider using weighting or analysis of nonresponse bias in the presence of low response rates. Lastly, in surveys of smaller societies, survey data analysis may utilize finite population correction when estimating standard errors of estimates, as many survey samples in our review represented significant proportions of the total society membership. Statistical guidelines on this technique recommend using a finite population correction, which deflates the standard error in proportion to the population fraction surveyed, when the sample size is more than 5% of the total population.³⁰ These recommendations could be incorporated into the development and evaluation of survey research for future studies in this area.

In summary, many surveys of anesthesiology professional societies have been conducted to study clinical, educational, and professional topics in the field. Technology has increased the ease of survey administration, as many survey invitations are distributed to all society members via email to complete on-line, with repeat reminders sent as often as once a month. Notwithstanding the ease of survey administration, response rates were often low even in published survey research. Investigators’ use of reminders and other aspects of survey methodology were not associated with higher response rates in published surveys of anesthesia professional societies. Furthermore, despite a high variability in response rates, the survey response rate was not associated with article impact as measured by article citations, Altmetric score, journal impact factor, or references in society publications, textbooks, or news media. Improving reporting of survey methodology and validating techniques for increasing the response rate specifically among members of anesthesiology professional societies may aid in increasing the quality of survey research in this area, and improve understanding of possible sources of nonresponse bias. Our findings provide a baseline for initiatives to improve survey research in anesthesiology professional societies, and a point of comparison for readers or reviewers assessing the quality of surveys in this population.

Acknowledgment

This review was presented in part at the Society for Pediatric Anesthesia 2018 Annual Meeting, March 24, 2018, Phoenix, AZ, USA.

Disclosure

The authors report no conflicts of interest in this work and no external funding.