The effect of rater training on scoring performance and scale-specific expertise amongst occupational therapists participating in a multicentre study: a single-group pre–post-test study

References

• Kobak KA. Inaccuracy in clinical trials: effects and methods to control inaccuracy. Curr Alzheimer Res 2010;7:637–41.
   PubMed Web of Science ®Google Scholar
• Kobak KA, Engelhardt N, Williams JB, Lipsitz JD. Rater training in multicenter clinical trials: issues and recommendations. J Clin Psychopharmacol 2004;24:113–17.
   PubMed Web of Science ®Google Scholar
• Wolfe EW. Methods for monitoring rating quality: Current practices and suggested changes (white paper). Austin (TX): Pearson. Available from: http://researchnetwork.pearson.com/wp-content/uploads/Wolfe_MethodsForMonitoring_May2014-2.pdf [last accessed 8 Dec 2014].
   Google Scholar
• Myford CM, Wolfe EW. Detecting and measuring rater effects using many facet Rasch measurement: part I. J Appl Meas 2003;4:386–422.
   PubMedGoogle Scholar
• Yeates P, O’Neill P, Mann K, Eva KW. Seeing the same thing differently. Mechanisms that contribute to assessor differences in directly-observed performance assessments. Adv Health Sci Educ Theory Pract 2013;18:325–41.
   PubMed Web of Science ®Google Scholar
• MacKenzie DE, Westwood DA. Observation patterns of dynamic occupational performance. Can J Occup Ther 2013;80:92–100.
   PubMed Web of Science ®Google Scholar
• Laver Fawcett A. Principles of assessment and outcome measurement for occupational therapists and physiotherapists: theory, skills and application. Chichester: John Wiley & Sons Ltd.; 2007.
   Google Scholar
• Hansen T, Lambert HC, Faber J. Content validation of a Danish version of “The McGill Ingestive Skills Assessment” for dysphagia management. Scand J Occup Ther 2011;18:282–93.
   PubMed Web of Science ®Google Scholar
• Hansen T, Lambert HC, Faber J. Validation of the Danish version of the McGill Ingestive Skills Assessment using classical test theory and the Rasch model. Disabil Rehabil 2012;34:859–68.
   PubMed Web of Science ®Google Scholar
• Hansen T. Further validation of the Danish version of the McGill Ingestive Skills Assessment (MISA-DK): Rasch analysis. Int J Ther Rehabil 2014;21:26–34.
   Google Scholar
• Hansen T. Lambert HC, Faber J. Reliability of the Danish version of the McGill Ingestive Skills Assessment for observation-based measures during meals. Scand J Occup Ther 2012;19:488–96.
   PubMed Web of Science ®Google Scholar
• Streiner DL, Norman GR. Health Measurements Scales. A practical guide to their development and use. 4th ed. Oxford: Oxford University Press; 2008.
   Google Scholar
• Feldman M, Lazzara EH, Vanderbilt AA, DiazGranados D. Rater training to support high-stakes simulation-based assessments. J Contin Educ Health Prof 2012;32:279–86.
   PubMed Web of Science ®Google Scholar
• Helker H, Stange C. Innovations, competencies, development. Rater training manual. 2013. Available from: http://incode-eu.eu/uploads/media/Manual_Rater_Training_INCODE.pdf [last accessed 8 Dec 2014].
   Google Scholar
• Roch SG, Woehr DJ, Mishra V, Kieszczynska U. Rater training revisited: an updated meta-analytic review of frame-of-reference training. J Occup Organ Psychol 2012;85:370–95.
   Web of Science ®Google Scholar
• Lambert HC, Gisel EG, Groher ME, et al. Psychometric testing of the McGill Ingestive Skills Assessment. Am J Occup Ther 2006;60:409–19.
   PubMed Web of Science ®Google Scholar
• Ministerial declaration of the Bachelor of Science in Occupational Therapy, BEK no 832 of August 13 2008 (Bekendtgørelse om uddannelsen til professionsbachelor i ergoterapi, BEK nr 832 af 13/08/2008). Available from: https://www.retsinformation.dk/forms/R0710.aspx?id=121015 [last accessed 15 Nov 2014].
   Google Scholar
• Clark GF, Avery-Smith W, Wold LS, et al. Eating and Feeding Task Force; Commission on Practice (AOTA). Specialized knowledge and skills in feeding, eating, and swallowing for occupational therapy practice. Am J Occup Ther 2007;61:686–700.
   PubMedGoogle Scholar
• Song T, Wolfe EW, Hahn L, et al. Relationship between rater background and rater performance. Austin (TX): Pearson. Available from: http://researchnetwork.pearson.com/wp-content/uploads/Song_RaterBackground_04_21_2014.pdf [last accessed 11 Dec 2014].
   Google Scholar
• Cusick A, Vasquez M, Knowles L, Wallen M. Effect of rater training on reliability of Melbourne assessment of unilateral upper limb function scores. Dev Med Child Neurol 2005;47:39–45.
   PubMed Web of Science ®Google Scholar
• Portney IG, Watkins MP. Foundations of clinical research. Application to practice. 3rd ed. Upper Saddle River (NJ): Pearson & Prentice Hall; 2009.
   Google Scholar
• Hansen TS, Jakobsen D. A decision-algorithm defining the rehabilitation approach: “Facial oral tract therapy1”. Disabil Rehabil 2010;32:1447–60.
   PubMed Web of Science ®Google Scholar
• Lov om videnskabsetisk behandling af sundhedsvidenskabelige forskningsprojekter, LOV nr 593 af 14/06/2011 [The Act on Research Ethics Review of Health Research Projects, Act no. 593 of 14 June 2011]. The ministry of Health and prevention. Available from: https://www.retsinformation.dk/Forms/R0710.Aspx?id=137674#Not1 [last accessed 11 Aug 2015].
   Google Scholar
• Lov om behandling af personoplysninger, LOV nr 429 af 31/05/2000 [The Act on Processing of Personal Data (Act No. 429 of 31 May 2000]. The Ministry of Justice. Available from: https://www.retsinformation.dk/Forms/R0710.Aspx?id=828 [last accessed 11 Aug 2015]
   Google Scholar
• Anderson LW, Krathwohl DR, eds. A taxonomy for learning, teaching and assessing: a revision of Bloom's Taxonomy of educational objectives: Complete edition. New York: Addison Wesley Longman, Inc.; 2001.
   Google Scholar
• Rocca KA. Student participation in the college classroom: an extended multidisciplinary literature review. Commun Educ 2010;59:185–213.
   Google Scholar
• Jeffries C, Maeder DW. Using vignettes to build and assess teacher understanding of instructional strategies. Prof Educ 2004;27:17–28.
   Google Scholar
• Chevalier A, Fielding A. An introduction to anchoring vignettes. J R Stat Soc Ser A Stat Soc 2011;174:569–74.
   Web of Science ®Google Scholar
• Cash AH, Hamre BK, Pianta RC, Meyers SS. Rater calibration when observational assessment occurs at large scales: degree of calibration and characteristics of raters associated with calibration. Early Child Res Quart 2012;27:529–42.
   Web of Science ®Google Scholar
• Graham M, Milanowski A, Miller J. Measuring and promoting inter-rater agreement of teacher and principal performance ratings. Washington (DC): Center for Educator Compensation Reform. Available from: http://files.eric.ed.gov/fulltext/ED533704.pdf [last accessed 8 Dec 2014].
   Google Scholar
• Kottner J, Audige L, Brorson S, et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol 2011;64:96–110.
   PubMed Web of Science ®Google Scholar
• Cicchetti DV. The precession of reliability and validity estimates re-visited: distinguishing between clinical and statistical significance of sample size requirements. J Clin Exp Neuropsychol 2001;23:695–700.
   PubMed Web of Science ®Google Scholar
• Altman DG. Practical statistics for medical research. New York: Chapman and Hall/CRC Press; 1991.
   Google Scholar
• Vejleskov H. Udvalgte Piaget-tekster – uddrag fra Piagets værker med indledende kommentarer. [Featured Piaget texts – excerpts from Piaget's works with introductory comments]. Copenhagen: Danish Psychological Publishing House; 1999.
   Google Scholar
• Kobak K, Lipsitz JD, Williams JB, et al. A new approach to rater training and certification in a multi-center clinical trial. J Clin Psychopharmacol 2005;25:407–12.
   PubMed Web of Science ®Google Scholar
• Jeglic E, Kobak KA, Engelhardt N, et al. A novel approach to the rater training and certification in multinational trials. Int Clin Psychopharmacol 2007;22:187–91.
   PubMed Web of Science ®Google Scholar
• Eckes T. Many-facet Rasch measurement. In: Takala S, ed. Reference supplement to the manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (Section H). Strasbourg: Council of Europe/Language Policy Division; 2009.
   Google Scholar
• Williams RG, Klamen DA, McGaghie WC. Cognitive, social, and environmental sources of bias in clinical competence ratings. Teach Learn Med 2003;15:270–92.
   PubMed Web of Science ®Google Scholar
• George PP, Papachristou N, Belisario JM, et al. Online eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health 2014;4:010406.
   PubMed Web of Science ®Google Scholar
• Popp D, Mallinckrodt CH, Williams JBW, Detke MJ. The unreliability of reliability statistics: a primer on calculating interrater reliability in cns clinical trials. Neuropsychopharmacology 2012;38:S134–5.
   Google Scholar