Rating Methodological Quality: Toward Improved Assessment and Investigation

Abstract

Assessing methodological quality is considered essential in deciding what investigations to include in research syntheses and in detecting potential sources of bias in meta-analytic results. Quality assessment is also useful in characterizing the strengths and limitations of the research in an area of study. Although numerous instruments to measure research quality have been developed, they have lacked empirically-supported components. In addition, different summary quality scales have yielded different findings when they were used to weight treatment effect estimates for the same body of research. Suggestions for developing improved quality instruments include: distinguishing distinct domains of quality, such as internal validity, external validity, the completeness of the study report, and adherence to ethical practices; focusing on individual aspects, rather than domains of quality; and focusing on empirically-verified criteria. Other ways to facilitate the constructive use of quality assessment are to improve and standardize the reporting of research investigations, so that the quality of studies can be more equitably and thoroughly compared, and to identify optimal methods for incorporating study quality ratings into meta-analyses.

Keywords:

• quality
• methodology
• bias
• meta-analysis
• evidence-based medicine
• systematic reviews

This work was supported by National Institute on Alcohol Abuse and Alcoholism grant AA08689, the Department of Veterans Affairs (VA) Quality Enhancement Research Initiative, and the VA Mental Health Strategic Healthcare Group. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. We thank Lanette Raymond and Paul Wortman for valuable comments on an earlier version of this manuscript.

Both authors made significant contributions to writing and editing the manuscript.

Notes

West, S., King, V., Carey, T. S., et al. (2002) Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290–97–0011). AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality