Not all systematic reviews are created equal: Considerations for appraisal

References

• Agency for Healthcare Research and Quality . 2002 . Criteria for determining disability in speech–language disorders. Available at: http://www.ahrq.gov/clinic/epcsums/spdissum.htm (accessed 13 September, 2007)
   Google Scholar
• Auperin , A , Pignon , J-P and Poynard , T . 1997 . Review article: Critical review of meta-analyses of randomized clinical trials in hepatogastroenterology . Aliment Pharmacol Ther , 11 : 215 – 225 .
   PubMed Web of Science ®Google Scholar
• Bochner , S , Center , Y , Chapparo , C and Donelly , M . 1999 . How effective are programs based on conductive education? A report of two studies . Journal of Intellectual & Development Disability , 24 : 227 – 242 .
   Web of Science ®Google Scholar
• Bopp , KD , Brown , KE and Mirenda , P . 2004 . Speech–language pathologists’ role in the delivery of positive behavior support for individuals with developmental disabilities . American Journal of Speech-Language Pathology , 13 : 5 – 19 .
   PubMed Web of Science ®Google Scholar
• Brady , N . 2007 . Data seem to support manual sign interventions for children with autism, but more high-quality research is needed . Evidence-Based Communication Assessment and Intervention , 1 : 16 – 17 .
   Google Scholar
• Campbell , PH , Milbourne , S , Dugan , LM and Wilcox , MJ . 2006 . A review of evidence on practices for teaching young children to use assistive technology devices . Topics in Early Childhood Special Education , 26 : 3 – 13 .
   Web of Science ®Google Scholar
• Choi , PT-L , Halpern , SH , Malik , N , Jadad , AR , Tramèr , MR and Walder , B . 2001 . Examining the evidence in anesthesia literature: A critical appraisal of systematic reviews . Anesthesia and Analgesia , 92 : 700 – 709 .
   PubMed Web of Science ®Google Scholar
• Clarke , M . 2000 . “ The QUORUM statement ” . In Lancet Vol. 26 , 756 – 757 .
   Google Scholar
• Cohen , J . 1960 . A coefficient of agreement for nominal scales . Educational and Psychological Measureement , 20 : 37 – 46 .
   Web of Science ®Google Scholar
• Cook , DJ , Sackett , DL and Spitzer , WO . 1995 . Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis . Journal of Clinical Epidemiology , 48 : 167 – 171 .
   PubMed Web of Science ®Google Scholar
• Cook , DJ , Mulrow , CD and Haynes , RB . 1997 . Synthesis of best evidence for clinical decisions . Annals of Internal Medicine , 126 : 376 – 380 .
   PubMed Web of Science ®Google Scholar
• Cook , TD , Cooper , H , Cordray , DS , Hartmann , H , Hedges , LV Light , RJ . 1992 . Meta-analysis for explanation: A casebook , New York : Russell Sage Foundation .
   Google Scholar
• Cooper , H and Hedges , LV . 1994 . The handbook of research synthesis , New York : Russell Sage Foundation .
   Google Scholar
• Darrah , J , Watkins , B , Chen , L and Bonin , C . 2003 . Effects of conductive education intervention for children with a diagnosis of cerebral palsy: An AACPDM evidence report. , Rosemont, IL, USA : American Academy for Cerebral Palsy and Developmental Medicine (AACPDM), AACPDM Evidence Report:34 .
   Google Scholar
• DeCoster , J . 2005 . “ Meta-analysis. ” . In Encyclopedia of social measurement , Edited by: Kempf-Leonard , K . Vol. 2 , 683 – 688 . San Diego, CA : Academic Press .
   Google Scholar
• Dollaghan , CA . 2004 . Evidence-based practice in communication disorders: what do we know, and when do we know it? . Journal of Communication Disorders , 37 : 391 – 400 .
   PubMed Web of Science ®Google Scholar
• Eagly , A , H and Wood , W . 1994 . “ Using research to plan future research. ” . In The handbook of research synthesis , Edited by: Cooper , H and Hedges , LV . 485 – 500 . New York : Russell Sage Foundation .
   Google Scholar
• Egger , M and Davey Smith , G . 1998 . Meta-analysis: Bias in location and selection of studies . British Medical Journal , 316 : 61 – 66 .
   PubMed Web of Science ®Google Scholar
• Egger , M and Smith , GD . 2001 . “ Principles of and procedures for systematic reviews. ” . In Systematic reviews in health care. . , 2nd ed , Edited by: Egger , M , Smith , GM and Altman , DG . 23 – 42 . London : British Medical Journal .
   Google Scholar
• Egger , M , Zellweger-Zahner , T , Schneider , M , Junker , C , Lengeler , C and Antes , G . 1997 . Language bias in randomised controlled trials published in English and German . Lancet , 350 : 326 – 329 .
   PubMed Web of Science ®Google Scholar
• Eysenck , HJ . 1978 . An exercise in mega-silliness . American Psychology , 33 : 517
   Web of Science ®Google Scholar
• Fleiss , JL . 1971 . Measuring nominal scale agreement among many raters . Psychological Bulletin , 76 : 378 – 382 .
   Web of Science ®Google Scholar
• Granlund , M . 2007 . “ Presents characteristics of AAC interventions for students with severe disabilities, but judgements about effectiveness do not follow from methodology [Abstract]. ” . In Evidence-Based Communication Assessment and Intervention, Vol. 1 , 67 – 68 .
   Google Scholar
• Abstract of Snell , M , Chen , L-H and Hoover , K . 2007 . “ Teaching augmentative communication to students with severe disabilities: A review of intervention research 1997–2003. ” . In Research and Practice for Persons with Severe Handicaps Vol. 31 , 203 – 214 .
   Google Scholar
• Grégoire , G , Derderian , F and LeLorier , J . 1995 . Selecting the language of the publications included in a meta-analysis: Is there a Tower of Babel bias? . Journal of Clinical Epidemiology , 48 : 159 – 163 .
   PubMed Web of Science ®Google Scholar
• Guyatt , G and Rennie , D . 2002 . “ Users’ guide to the medical literature ” . In Essentials of evidence-based clinical practice , Chicago : AMA Press .
   Google Scholar
• Herder , C , Howard , C , Nye , C and Vanryckeghem , M . 2006 . Effectiveness of behavioral stuttering treatment: A systematic review and meta-analysis . Contemp Iss Comm Sci Dis , 33 : 61 – 73 .
   Google Scholar
• Higginbotham , DJ . 2003 . “ Formulating research questions ” . In The efficacy of augmentative and alternative communication: Toward evidence-based practice , Edited by: Schlosser , RW . 43 – 57 . San Diego : Academic Press .
   Google Scholar
• Higgins , JPT and Green , S . 2006 . Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September, 2006]. Available at: http://www.cochrane.org/resources/handbook/hbook.htm (accessed 6th October, 2006)
   Google Scholar
• Humphris , D , Littlejohns , P , Victor , C , O’Halloran , P and Peacock , J . 2000 . Implementing evidence-based practice: Factors that influence the use of research evidence by occupational therapists . Br J Occup Ther , 63 : 516 – 522 .
   Google Scholar
• Hyung Bok Yoo , H and Quebuz , TT . 2004 . Locating and selecting appraisal studies for reviews . Chest , 125 : 798
   PubMedGoogle Scholar
• Jackson , GB . 1980 . Methods for integrative reviews . Review of Educational Research , 50 : 438 – 460 .
   Web of Science ®Google Scholar
• Jüni , P , Holenstein , F , Sterne , J , Bartlett , C and Egger , M . 2002 . Direction and impact of language bias in meta-analyses of controlled trials: empirical study . International Journal Of Epidemiology , 31 : 115 – 123 .
   PubMed Web of Science ®Google Scholar
• Landis , JR and Koch , GG . 1977 . The measurement of observer agreement for categorical data . Biometrics , 33 : 159 – 174 .
   PubMed Web of Science ®Google Scholar
• Lauer , JM and Asher , JW . 1988 . Composition research: Empirical designs , New York : Oxford University Press .
   Google Scholar
• Law , M . 2002 . Evidence-based rehabilitation: A guide to practice , Thoroughfare, NJ : Slack Inc .
   Google Scholar
• Law , J . 2007 . Behavioral stuttering treatments are effective but no one treatment approach is more effective over other treatment approaches . Evidence-based Communication Assessment and Intervention , 1 : 14 – 15 .
   Google Scholar
• Law , M and Plunkett , C . 2006 . Grading study quality in systematic reviews . Contemp Iss Comm Sci Dis , 33 : 28 – 36 .
   Google Scholar
• Light , R and Pillemer , D . 1984 . Summing up: The science of reviewing research , Cambridge, MA : Harvard University Press .
   Google Scholar
• Lipsey , MW and Wilson , DB . 2001 . Practical meta-analysis , Thousand Oaks, CA : Sage Publications .
   Google Scholar
• Maher , CG , Sherrington , C , Herbert , RD , Moseley , AM and Elkins , M . 2003 . Reliability of the PEDro scale for rating quality of randomized controlled trials . Physical Theraphy , 83 : 713 – 721 .
   PubMed Web of Science ®Google Scholar
• Mancil , GR . 2006 . Functional communication training: A review of the literature related to children with autism . Education and Training in Development Disabilities , 41 : 213 – 224 .
   Web of Science ®Google Scholar
• McAuley , L , Pham , B , Tugwell , P and Moher , D . 2000 . Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? . The Lancet , 356 : 1228 – 1231 .
   PubMed Web of Science ®Google Scholar
• Meline , T . 2006 . Selecting studies for systematic review: Inclusion and exclusion criteria . Contemp Iss Comm Sci Dis , 33 : 21 – 27 .
   Google Scholar
• Millar , D , Light , JC and Schlosser , RW . 2006 . The impact of augmentative and alternative communication intervention on the speech production of individuals with developmental disabilities: A research review . Journal of Speech Language and Hearing Research , 49 : 248 – 264 .
   PubMed Web of Science ®Google Scholar
• Moher , D , Cook , DJ , Eastwood , S , Olkin , I , Rennie , D and Stroup , DF . 1999 . Improving the quality of reporting of meta-analysis of randomized controlled trials: The QUOROM statement . Lancet , 354 : 1896 – 1900 .
   PubMed Web of Science ®Google Scholar
• Moher , D , Pham , B , Klassen , TP , Schulz , KF , Berlin , JA , Jadad , AR and Liberati , A . 2000 . What contributions do languages other than English make on the results of meta-analyses? . Journal of Clinical Epidemiology , 53 : 964 – 972 .
   PubMed Web of Science ®Google Scholar
• Moher , D , Schulz , KF and Altman , DG . 2001 . The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials . Lancet , 357 : 1191 – 1194 .
   PubMed Web of Science ®Google Scholar
• Moher , D , Tetzlaff , J , Tricco , AC , Sampson , M and Altman , DG . 2007 . Epidemiology and reporting characteristics of systematic reviews . PLoS Medicine , 4 : e78
   PubMed Web of Science ®Google Scholar
• Montori , VM , Swiontkowski , MF and Cook , DJ . 2003 . Methodologic issues in systematic reviews and meta-analyses . Clinical Orthopaedics and Related Research , 413 : 43 – 54 .
   PubMed Web of Science ®Google Scholar
• Mortaz Hejri , S . 2003 . Critical Appraisal tools for systematic reviews. Available at http://ssrc.tums.ac.ir/SystematicReview/Appraisal-Tools.asp (accessed 13 September, 2007)
   Google Scholar
• Oxman , AD , Cook , DJ and Guyatt , GH . 1994 . Users’ guide to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group . Journal ofAmerican Medical Association , 272 : 1367 – 1371 .
   PubMed Web of Science ®Google Scholar
• Perepletchikova , F and Kazdin , AE . 2005 . Treatment integrity and therapeutic change: Issues and research recommendations . Clinical Psychology , 12 : 365 – 383 .
   Google Scholar
• Peterson , L , Homer , AL and Wonderlich , SA . 1982 . The integrity of independent variables in behavior analysis . Journal Of Applied Behavior Analysis , 15 : 477 – 492 .
   PubMed Web of Science ®Google Scholar
• Petticrew , M and Roberts , H . 2006 . Systematic reviews in the social sciences: A practical guide , Malden, MA : Blackwell Publishing Co .
   Google Scholar
• Reilly , S and Oates , J . 2003 . Evidence-based practice in speech pathology , London : Whurr .
   Google Scholar
• Rosenthal , R . 1978 . The "file drawer problem" and tolerance for null results . Psychological Bulletin , 86 : 638 – 641 .
   Web of Science ®Google Scholar
• Rothstein , H , Sutton , AJ and Borenstein , M . 2005 . Publication bias in meta-analysis: Prevention, assessment and adjustments , UK : John Wiley & Sons .
   Google Scholar
• Sackett , DL and Gent , M . 1979 . Controversy in counting and attributing events in clinical trials . New England Journal of Medicine , 301 : 1410 – 1412 .
   PubMed Web of Science ®Google Scholar
• Sackett , DL , Haynes , RB and Tugwell , P . 1985 . Clinical epidemiology: A basic science of clinical medicine , Boston : Little, Brown .
   Google Scholar
• Sackett , DL , Straus , SE , Richardson , WS , Rosenberg , W and Haynes , RB . 2000 . Evidence-based medicine: How to practice and teach EBM , 2nd , Edinburgh : Churchill Livingstone .
   Google Scholar
• Sacks , HS , Reitman , D , Pagano , D and Kupelnick , B . 1996 . Meta-analysis: An update . Mount Sinai Journal Of Medicine , 63 : 216 – 224 .
   PubMedGoogle Scholar
• Schlesselman , JJ and Collins , JA . 2003 . Evaluating systematic reviews and meta-analyses . Seminars in Reproductive Medicine , 21 : 95 – 105 .
   PubMed Web of Science ®Google Scholar
• Schlosser , RW . 1999 . Social validation of interventions in augmentative and alternative communication . Augment Altern Commun , 15 : 234 – 247 .
   Google Scholar
• Schlosser , RW . 2002 . On the importance of being earnest about treatment integrity . Augment Altern Comm , 18 : 36 – 44 .
   Google Scholar
• Schlosser , RW . 2003a . The efficacy of augmentative and alternative communication: Toward evidence-based practice , San Diego, CA : Academic Press .
   Google Scholar
• Schlosser , RW . 2003b . “ Synthesizing efficacy research in AAC ” . In The efficacy of augmentative and alternative communication: Towards evidence-based practice , Edited by: Schlosser , RW . 230 – 258 . San Diego, CA : Academic Press .
   Google Scholar
• Schlosser , RW . 2007 . Training of conversation partners of children with cerebral palsy appears effective for common communication targets but methodological concerns render these findings suggestive . Evidence Based Communication Assessement and Intervention , 1 : 12 – 13 .
   Google Scholar
• Schlosser , RW and Goetze , H . 1992 . “ Effectiveness and treatment validity of interventions addressing self-injurious behavior: From narrative reviews to meta-analysis ” . In Advances in learning and behavioral disabilities , Edited by: Scruggs , TE and Mastropieri , MA . Vol. 7 , 135 – 175 . Greenwich, CT : JAI Press, Inc .
   Google Scholar
• Schlosser , RW and Lee , D . 2000 . Promoting generalization and maintenance in augmentative and alternative communication: A meta-analysis of 20 years of effectiveness research . Augment Altern Comm , 16 : 208 – 227 .
   Google Scholar
• Schlosser , RW and O’Neil-Pirozzi , TM . 2006 . Problem formulation in evidence-based practice and systematic reviews . Contemp Iss Comm Sci Dis , 33 : 5 – 10 .
   Google Scholar
• Schlosser , RW and Raghavendra , P . 2004 . Evidence-based practice in augmentative and alternative communication . Augment Altern Comm , 20 : 1 – 21 .
   Google Scholar
• Schlosser , RW , Wendt , O , Angermeier , K and Shetty , M . 2005 . Searching for and finding evidence in augmentative and alternative communication: Navigating a scattered literature . Augment Altern Comm , 21 : 233 – 255 .
   Web of Science ®Google Scholar
• Schwartz , JB and Nye , C . 2006 . A systematic review, synthesis, and evaluation of the evidence for teaching sign language to children with autism . EBP Briefs , 1 : 1 – 17 .
   Google Scholar
• Scruggs , TE . 1992 . “ Single-subject research methodology in the study of learning and behavioral disorders: Design, analysis, and synthesis ” . In Advances in learning and behavioral disabilities , Edited by: Scruggs , TE and Mastropieri , M. A. Vol. 7 , 223 – 248 . Greenwich, CT : TE .
   Google Scholar
• Scruggs , TE and Mastropieri , MA . 1998 . Summarizing single-subject research: Issues and applications . Behaviour Modification , 22 : 221 – 242 .
   PubMed Web of Science ®Google Scholar
• Slavin , RE . 1987 . “ Best-evidence synthesis: An alternative to meta-analytic and traditional reviews ” . In Evaluation studies: Review annual 12 , Edited by: Shadish , W. R. and Reichardt , C. S. 667 – 673 . Thousand Oaks, CA : Sage Publications .
   Google Scholar
• Snell , ME , Chen , L-H and Hoover , K . 2006 . Teaching augmentative and alternative communication to students with severe disabilities: A review of intervention research, 1997–2003 . Res Pract Pers Sev Disabil , 31 : 203 – 214 .
   Web of Science ®Google Scholar
• Straus , SE , Richardson , WS , Glasziou , P and Haynes , RB . 2005 . Evidence-based medicine: How to practice and teach EBM , 3rd , Edinburgh : Elsevier Science .
   Google Scholar
• Turner , HM and Bernard , RM . 2006 . Calculating and synthesizing effect sizes . Contemp Iss Comm Sci Dis , 33 : 42 – 55 .
   Google Scholar
• Verhagen , AP , de Vet , HC , de Bie , RA , Kessels , AG , Boers , M , Bouter , LM and Knipschild , PG . 1998 . The Delphi list: A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus . Journal of Clinical Epidemiology , 51 : 1235 – 1241 .
   PubMed Web of Science ®Google Scholar
• Wendt , O . 2006 . The effectiveness of augmentative and alternative communication for children with autism: A meta-analysis of intervention outcomes [dissertation]. , West Lafayette, IN : Purdue University .
   Google Scholar
• Wendt , O . 2007 . Recommended practices for teaching assistive technology use to infants and young children with low incidence disabilities seem to have little empirical support but methodological concerns limit the validity of this review . Evidence Based Communication Assessment and Intervention , 1 : 60 – 62 .
   Google Scholar
• West , S , King , V , Carey , T , S , Lohr , K , N , McKoy , N , Sutton , S , F and Lux , L . 2002 . “ Systems to Rate the Strength of Scientific Evidence. ” . In Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute–University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ Publication No. 02-E016. , Rockville, MD : Agency for Healthcare Research and Quality .
   Google Scholar
• White , HD . 1994 . “ Scientific communication and literature retrieval ” . In The handbook of research synthesis , Edited by: Cooper , H. and Hedges , L. V. 41 – 55 . New York : Russell Sage Foundation .
   Google Scholar
• Zipoli , RP and Kennedy , M . 2005 . Evidence-based practice among speech–language pathologists: Attitudes, utilization, and barriers . American Journal of Speech-Language Pathology , 14 : 208 – 220 .
   PubMed Web of Science ®Google Scholar