Intercoder Reliability for Use in Qualitative Research and Evaluation

Abstract

Intercoder reliability can increase trustworthiness, accuracy, rigor, collaboration, and power sharing in qualitative research. Though not every qualitative design can utilize intercoder reliability, this article highlights how positivist qualitative research, community-based participatory research, and participatory evaluation all strengthen when intercoder reliability is sought among coding teams. Basic calculations for intercoder reliability can be completed manually or with software, making it an accessible metric to inform counseling research and evaluation, benefitting counselor education and practice. Interpreting metrics like percent agreement or kappa statistic largely depends on data context and potential impact of the study, which warrants researchers and evaluators to assess and understand the consequences of their work. Using ICR with clinical data such as interviews and other qualitative sources can be utilized to teach trainees the method in counseling research and evaluation and help make visible the important contributions of qualitative research using collaborative teams in counselor education.

SIGNIFICANCE STATEMENT

Intercoder reliability is a method in qualitative coding analysis that helps multiple coders increase accuracy in their results based on agreement in their coding. The method is particularly important in qualitative research and evaluation like positivist qualitative research, community-based participatory research, and participatory evaluation, where rigor, power sharing, and participation, respectively, are valued among the research teams.

Keywords:

• Intercoder reliability
• qualitative analysis
• positivist qualitative research
• community-based participatory research
• participatory evaluation

Acknowledgements

We would like to acknowledge the many community members and fellow researchers who served as co-researchers with us throughout our careers. Each of you shaped our thinking and belief in the power of collaboration in research and evaluation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Monica L. Coleman

Monica L. Coleman, MS, CRC, NCC is a community-powered researcher and evaluator and a doctoral candidate in Counselor Education and Supervision at the University of Mississippi. Her research and evaluation centers the leadership, voice, and power of those most impacted by structural oppression.

Moira Ragan

Moira Ragan is the Director of Research and Evaluation at Ripple Effect Communications and is completing her PhD in Educational Research and Evaluation at the Patton College of Education, Ohio University. Her background is in methodology and psychometrics and her work is largely in healthcare and education.

Tahani Dari

Tahani Dari, PhD, LPC (MI), NCC is an assistant professor in the Department of Counseling at the University of Detroit Mercy. Her research focuses on community based participatory research and research interests address topics in research/evaluation methods, clinical/school counseling, counselor education as well as clinical supervision.