Estimating Interrater Reliability of Examiner Scoring for a State Quality Award

Abstract

Examiner scores for two years of a state quality award were analyzed by sector to estimate interrater reliability. The intraclass correlation coefficients (ICC), ICC(1,1) and ICC(2,1), were chosen as the statistics to estimate reliability to enable the researchers to generalize results from specific examiner teams to the larger pool of examiners, thus providing an assessment of the overall scoring effectiveness in the state quality award. These forms of ICC address inter-changeability of examiners—that is, they provide an estimate of the reliability of the pool of examiners no matter which individual examiners were selected to evaluate applicants. Interrater reliability, as shown by ICC values, ranged from low to moderate, with significant values ranging from 0.18 to 0.58. These low correlation coefficient values were likely due to the inferential limitations of the field data used as much as any lack of consistency among examiners.

• quality award examiners
• rater consistency
• rater reliability

Additional information

Notes on contributors

Garry D. Coleman

Garry D. Coleman is an assistant professor of industrial engineering at the University of Tennessee. His research interests include examining the effectiveness of organizational assessments and third-party review, developing approaches for evaluating and improving organizational performance measurement systems, and linking each of these areas to a continual process of planning, implementing, measuring, and improvement. He has studied how organizations use planning and measurement to improve performance since he joined The Virginia Productivity Center (later known at The Performance Center) in 1986. He received his doctorate from Virginia Tech in industrial and systems engineering. He also received his master's degree in industrial engineering and operations research and a bachelor's degree in mining engineering from Virginia Tech. He is a senior member of the Institute of Industrial Engineers, a member of ASQ, NSPE, ASEM, and SEMS, and a 2001 examiner for the Tennessee Quality Award. Coleman is a licensed professional engineer and a Fellow of the World Academy of Productivity Sciences. He can be reached at University of Tennessee, 411 B. H. Goethert Pkwy., Tullahoma, TN 37388–9700, or by e-mail at [email protected].

Eileen M. Van Aken

Eileen M. Von Aken is an assistant professor in the Grado Department of Industrial and Systems Engineering at Virginia Tech and director of the Enterprise Engineering Research Lab. Her research interests include performance measurement, organizational and work system assessment, organizational transformation, lean production, and team-based work system design. She was employed for seven years at the Center for Organizational Performance Improvement at Virginia Tech and for two years as a process/product engineer at AT&T Microelectronics. She received her bachelor's, master's, and doctorate degrees from Virginia Tech in industrial and systems engineering. She is a member of IIE, ASEM, and ASQ, and is on examiner and current vice chair of the board for the U. S. Senate Productivity and Quality Award for Virginia. Van Aken is a Fellow of the World Academy of Productivity Sciences.

Jianming Shen

Jianming Shen is a reliability engineer for Lennox Industries in Carrollton, Texas. He received his master's degree in industrial engineering from the University of Tennessee. He received a master's degree in mechanical engineering from the Shanghai Academy of Space Flight Technology and his bachelor's degree in mechanical engineering from Shanghai Jiaotong University in China. Prior to attending the University of Tennessee, he worked in the space industry for live years as a product system engineer. He is a member of ASQ and is an ASQ Certified Reliability Engineer and Certified Quality Engineer.