Medical education research study quality instrument: an objective instrument susceptible to subjectivity

ABSTRACT

Background: The medical education research study quality instrument (MERSQI) was designed to appraise medical education research quality based on study design criteria. As with many such tools, application of the results may have unintended consequences. This study applied the MERSQI to published medical education research identified in a bibliometric analysis. Methods: A bibliometric analysis identified highly cited articles in medical education that two authors independently evaluated using the MERSQI. After screening duplicate or non-research articles, the authors reviewed 21 articles with the quality instrument. Initially, five articles were reviewed independently and results were compared to ensure agreed upon understanding of the instrument items. The remainder of the articles were independently reviewed. Overall scores for the articles were analyzed with a paired samples t-test and individual item ratings were analyzed for inter-rater reliability. Results: There was a significant difference in mean MERSQI score between reviewers. Inter-rater reliability for MERSQI items labeled response rate, validity and outcomes were considered unacceptable. Conclusions: Based on these results there is evidence that MERSQI items can be significantly influenced by interpretation, which lead to a difference in scoring. The MERSQI is a useful guide for identifying research methodologies. However, it should not be used to make judgments on the overall quality of medical education research methodology in its current format. The authors make specific recommendations for how the instrument could be revised for greater clarity and accuracy.

KEYWORDS:

• Medical education
• research
• MERSQI
• quality
• bibliometric analysis

Background

Medical education research and scholarship submissions continue to proliferate. Many editors of medical education journals have noted the increased number of submissions they receive, which exploded during the recent pandemic. Over the years, there have been attempts to address the quality of educational research, particularly as it relates to published work, peer-review processes, and study design [1]. These did not specifically address educational research methodology, which critics have noted as the primary quality concern [1,2].

The MERSQI was designed to assess the quality of medical education research methodology and to offer guidance on medical education research design [2]. The MERSQI specifically focuses on the quality of experimental, quasi-experimental, and observational studies. Upon closer inspection, the MERSQI scores research studies on multiple categories, such study design, sampling institutions and response rate, type of data, data analysis, validity and outcomes [1]. Internal medicine and obstetrics and gynecology have tested its validity and usefulness in scoring literature [3,4]. Studies indicate that certain scores on the MERSQI are more likely to be published than others; specifically, average score for published studies was 10.7 (SD 2.5) compared to 9.0 (SD 2.4) for rejected studies [5].

Although helpful for researchers as a guide for study design, a score should not dictate if a study should be conducted. Strict adherence to standardized instruments such as MERSQI may result in innovative research being overlooked. The senior author (GLBD) participated in a research group that was using the MERSQI for that purpose.

Another example of how recommended standardization may be used inappropriately is reflective in the recommendations introduced by Artino et al. [6] for reporting survey studies. The senior author (GLBD) has experienced comments from journal submissions that rigidly accept these recommendations as absolutes, suggesting that every item on the checklist be met to be considered for revision. Other submissions that focused on program evaluation were mistaken as surveys simply because a popular web-based survey platform was used. When the reviewer saw the software, they indicated that our work needed to be rejected because we did not follow the published guidelines [6].

Survey research reporting offers one example of measurement that results in unintended consequences if use of the checklist guidelines strays from its intended use. Although considering quality in medical education research is important, too much weight placed on a score may result in similar unintended consequences. The goal of this study is to apply the MERSQI to highly cited medical education articles and to see if there are significant discrepancies in how articles are rated, as well as see if the MERSQI would rate these highly cited articles below the average published rating.

Materials and methods

The MERSQI was applied to articles identified in a bibliometric analysis [7]. The MERSQI was constructed using literature to guide the item language. The final MERSQI includes 10 items, which can be used to identify an overall score. However, the interpretation of scores should focus on item-specific codes as opposed to an overall assessment of quality [1,5].

Bibliometric analyses are used to identify the most frequently cited articles in a particular field of study [8]. Azar’s bibliometric analysis was selected because it identified medical education articles based on the number of citations and keywords, making for a comprehensive review [7]. The bibliometric analysis included non-research articles, which were removed from the sample. Azar’s results included 112 medical education articles that were screened for evaluation using MERSQI. The article included two lists and duplicates from each list were screened out. The final list of articles included 11 from List A and 10 from List B for a total of 21 articles.

The authors have experience as researchers in various settings. Although one of the authors is a medical student, his experiences in undergraduate college as well as medical school have provided ample opportunities to understand research methodologies. The senior author is an experienced educational researcher with over 25 years of experience conducting educational research. Therefore, the research team has appropriate experience to analyze educational research methods by applying the MERSQI in a systematic way.

The authors independently evaluated articles identified from Azar’s study [7] using the MERSQI. After completing five reviews, the authors met to ensure there was agreement about how to rate MERSQI items. For those items that were unclear, the research team discussed the article extensively and the MERSQI item to ensure mutual understanding of the MERSQI item. After this meeting, the researchers independently reviewed the remaining articles. The MERSQI form was set up in Qualtrics (Provo, UT) to collect the data.

For each item of the MERSQI, inter-rater reliability was calculated. For inter-rater reliability, a minimum of two reviews need to be conducted independently. We interpreted Cronbach’s alpha results as excellent ≥ .9, good = .8–.9, acceptable = .7–.8, fair = .6–.7, poor = .5–.6, <.5 unacceptable. Overall scores were analyzed to determine if they met the requirements for normal distribution. Data was normally distributed so the overall MERSQI scores were compared using independent samples t-test. Effect size was calculated using Cohen’s d (.2=small effect, .5=medium effect, .8=large effect) [9].

Results

Each MERSQI item was analyzed for inter-rater reliability. Four of the items had good or excellent reliability (α≥.8). MERSQI items that are classified as Response rate, Validity, and Outcome were all considered unacceptable (α <.5) (Table 1).

Table 1. Inter-rater reliability of MERSQI items.

MERSQI Category	Cronbach’s alpha*
Design:	.82
Institution:	.89
RR (Response rate):	.02
Data Type:	.77
Validity:	.15
Sophistication:	.93
Appropriate:	1.0
Outcome:	.43

*For most studies, Cronbach’s alpha may be interpreted in the following way:

≥.9 Excellent

.8–.9 Good

.7–.8 Acceptable

.6–.7 Fair

.5−.6 Poor

<.5 Unacceptable

A total MERSQI score was calculated by reviewer for each article (Table 2). A paired samples t-test identified a significant difference in mean MERSQI score between reviewer 1 (M = 9.83, SD = 2.25) and reviewer 2 (M = 11.14, SD = 2.34); t(21) = 3.71, p < .001, d = .81).

Table 2. Total MERSQI score by reviewer.

Article ID	Reviewer 1	Reviewer 2
A1 1 [10]	8.50	10.50
A1 17 [11]	12.00	12.00
A1 21 [12]	6.00	7.00
A1 24 [13]	7.50	7.00
A1 39 [14]	7.50	9.50
A1 46 [15]	11.50	10.50
A1 46 [16]	10.50	12.00
A1 47 [17]	8.50	9.00
A1 49 [18]	9.50	9.50
A1 5 [19]	8.50	13.00
A1 50 [20]	10.50	14.00
A2 15 [21]	7.50	8.50
A2 16 [22]	14.50	15.00
A2 23 [23]	13.50	14.50
A2 30 [24]	13.00	13.00
A2 42 [25]	11.00	11.50
A2 43 [26]	8.00	13.00
A2 43 [27]	8.50	8.50
A2 46 [28]	10.00	13.00
A2 50 [29]	8.50	11.00
A2 50 [30]	11.50	12.00

Discussion

Based on our findings, scoring highly cited medical education research using the MERSQI failed to achieve consistent results in scoring. Response Rate, Validity, and Outcomes fell well below an acceptable Cronbach's alpha for inter-rater reliability. Additionally, overall MERSQI scores were significantly different. This calls into question whether or not this particular scale is useful for determining the quality of educational research methodology.

For Response Rate, it was challenging to determine when reviewing the articles, especially for manuscripts that were meta-analyses. The meta-analysis does not specify the response rate so it comes down to the rater on if they count the collection of data as a response rate or as a lack of response rate recorded. Although meta-analyses and other literature reviews are considered medical education research, using MERSQI to rate them was challenging, particularly for the Response Rate. Clarifying what constitutes a response in such studies may improve the reproducibility of the response rate variable of the MERSQI.

Understanding of validity evidence has evolved over time [31]. For many reviewed studies, multiple sources of validity evidence were identified. However, we used a single-best option response, which may be counter to what the MERSQI creators envisioned [5]. Our poor inter-rater reliability may have been mitigated had we checked all items that applied and calculated a grand sum. Based on our works, we recommend the MERSQI be revised to include more options for validity evidence as well as revise the scoring such that multiple items can be selected.

The lack of inter-rater reliability for Outcomes fell into the same trap as the Validity item. There were multiple outcome types reported, making it difficult to determine what was best. Although not shown in our data, we discussed the outcome ratings and adjusted scores after reaching agreement, which provided a slight improvement in inter-rater reliability. This particular item should also be revised to indicate that multiple options may be selected.

The statistically significant differences in overall MERSQI scores with a large effect size most likely was the result of inaccurate item scores. Four articles ranged from below 9 or above 10. As stated previously, research has shown that the average published score was greater than 10, and the average rejected score was equal to or below 9 [5]. This highlights how differences in the interpretation of the MERSQI can lead to markedly different outcomes of being published or not if too much weight is placed on a particular checklist or scale.

Lastly, this data shows that using the MERSQI both raters graded 7 of the same articles from the bibliometric analysis below 9. If purely going off of MERSQI scores as a decision to accept, these articles may not have been published, and yet they are some of the most highly cited articles based on this bibliometric analysis [7].

There have been other published recommendations seeking to standardize reporting survey research and qualitative research. Artino et al. [6] introduced guidelines for reporting survey studies, which were previously mentioned. Additionally, O’Brien et al. [32] produced a similar article for qualitative research. Both articles attempt to offer some form of standardized reporting and were not suggesting their criteria be used as judgment on the quality of the study being reported. Similarly, although there have only been examples from meetings [3] uses the MERSQI to screen the merits of potential projects, we believe the MERSQI should be used to inform researchers of key components to include as they develop and report on their studies.

This study is limited in that we focused solely on one scale for analyzing already published articles. However, given that the MERSQI has been used as a means of determining the quality of medical education research [3,4], using the MERSQI to rate highly cited medical education research has uncovered some flaws with the instrument with regard to specific items. If an overall score continues to be used for screening purposes, revisions should be made to address the limitations posed by Response Rate, Validity, and Outcomes. It would also be beneficial to have additional scoring conducted using the MERSQI that involves multiple reviewers in order to conduct a generalizability study, which is more robust than inter-rater reliability [33].

Conclusion

Our study has shown discrepancies between users’ ratings of the MERSQI checklist. Therefore, it is important to use the MERSQI as a guideline for appraising and designing research but not a tool to screen. Undertaking educational research project decisions should employ a holistic approach, which the MERSQI could contribute.

Disclosure statement

No potential conflict of interest was reported by the author(s)

Additional information

Funding

The author(s) reported that there is no funding associated with the work featured in this article.