The use of intercoder reliability in qualitative interview data analysis in science education

References

• *Borgerding, L. A., and F. Kaya. 2019. “Preschool Children’s Ideas about Biological Adaptation during a Science Camp.” International Journal of Science Education 41 (17): 2410–2429. doi:10.1080/09500693.2019.1683911.
   Web of Science ®Google Scholar
• Braun, V., and V. Clarke. 2013. Successful Qualitative Research. London: Sage.
   Google Scholar
• Brock, R., and K. S. Taber. 2017. “The application of the microgenetic method to studies of learning in science education: characteristics of published studies, methodological issues and recommendations for future research”. Studies in Science Education 53 (1): 45–73.
   Web of Science ®Google Scholar
• Campbell, J. L., C. Quincy, J. Osserman, and O. K. Pedersen. 2013. “Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement.” Sociological Methods & Research 42 (3): 294–320. doi:10.1177/0049124113500475.
   Web of Science ®Google Scholar
• Chang, Y.-H., C.-Y. Chang, and Y.-H. Tseng. 2009. “Trends of Science Education Research: An Automatic Content Analysis.” Journal of Science Education and Technology 19 (4): 315–331. doi:10.1007/s10956-009-9202-2.
   Web of Science ®Google Scholar
• Cheung, K. K. C., and M. Winterbottom. 2021. “Students’ Integration of Textbook Representations into Their Understanding of Photomicrographs: Epistemic Network Analysis.” Research in Science & Technological Education 1–20. doi:10.1080/02635143.2021.1920382.
   Web of Science ®Google Scholar
• *Corpuz, E. D. G., and N. S. Rebello. 2019. “Refining Students’ Explanations of an Unfamiliar Physical Phenomenon-Microscopic Friction.” Research in Science Education 49 (5): 1177–1211. doi:10.1007/s11165-017-9650-2.
   Web of Science ®Google Scholar
• *Dare, Emily A., Elizabeth A. Ring-Whalen, and Gillian H. Roehrig. 2019. “Creating a Continuum of STEM Models: Exploring How K-12 Science Teachers Conceptualize STEM Education.” International Journal of Science Education 41 (12): 1701–1720. doi:10.1080/09500693.2019.1638531.
   Web of Science ®Google Scholar
• Feng, G. C. 2014. “Intercoder Reliability Indices: Disuse, Misuse, and Abuse.” Quality & Quantity 48 (3): 1803–1815. doi:10.1007/s11135-013-9956-8.
   Google Scholar
• Feng, G. C. 2015. “Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices.” Methodology 11 (1): 13–22. doi:10.1027/1614-2241/a000086.
   Web of Science ®Google Scholar
• *Fragkiadaki, G., M. Fleer, and K. Ravanis. 2019. “A Cultural-Historical Study of the Development of Children’s Scientific Thinking about Clouds in Everyday Life.” Research in Science Education 49 (6): 1523–1545. doi:10.1007/s11165-017-9665-8.
   Web of Science ®Google Scholar
• *Fu, G., and A. Clarke. 2019. “Individual and Collective Agencies in China’s Curriculum Reform: A Case of Physics Teachers.” Journal of Research in Science Teaching 56 (1): 45–63. doi:10.1002/tea.21467.
   Web of Science ®Google Scholar
• Glaser, B., and A. Strauss. 1967. The Discovery of Grounded Theory. London: Weidenfield & Nicolson.
   Google Scholar
• Gwet, K. L. 2014. Handbook of Inter-rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Raters. Advanced Analytics, LLC.
   Google Scholar
• Hallgren, K. A. 2012. “Computing Inter-rater Reliability for Observational Data: An Overview and Tutorial.” Tutorials in Quantitative Methods for Psychology 8 (1): 23. doi:10.20982/tqmp.08.1.p023.
   PubMedGoogle Scholar
• Hammer, D., and L. K. Berland. 2014. “Confusing Claims for Data: A Critique of Common Practices for Presenting Qualitative Research on Learning.” Journal of the Learning Sciences 23 (1): 37–46. doi:10.1080/10508406.2013.802652.
   Web of Science ®Google Scholar
• Hammersley, M., and P. Atkinson. 2007. Ethnography: Principles in Practice. London: Routledge.
   Google Scholar
• *Hecht, M., K. Knutson, and K. Crowley. 2019. “Becoming a Naturalist: Interest Development across the Learning Ecology.” Science Education 103 (3): 691–713. doi:10.1002/sce.21503.
   Web of Science ®Google Scholar
• Jansen, S., M.-C. P. J. Knippels, and W. R. Van Joolingen. 2019. “Assessing Students’ Understanding of Models of Biological Processes: A Revised Framework.” International Journal of Science Education 41 (8): 981–994. doi:10.1080/09500693.2019.1582821.
   Web of Science ®Google Scholar
• Khishfe, R. 2019. “The Transfer of Nature of Science Understandings: A Question of Similarity and Familiarity of Contexts.” International Journal of Science Education 41 (9): 1159–1180. doi:10.1080/09500693.2019.1596329.
   Web of Science ®Google Scholar
• Kier, M. W., and J. A. Chen. 2019. “Kindling the Fire: Fueling Preservice Science Teachers‘ Interest to Teach in High-needs Schools.” Science Education 103 (4): 875–899. doi:10.1002/sce.21520.
   Web of Science ®Google Scholar
• King, N. S., and R. M. Pringle. 2019. “Black Girls Speak STEM: Counterstories of Informal and Formal Learning Experiences.” Journal of Research in Science Teaching 56 (5): 539–569. doi:10.1002/tea.21513.
   Web of Science ®Google Scholar
• Krell, M., C. Walzer, S. Hergert, and D. Krüger. 2019. “Development and Application of a Category System to Describe Pre-Service Science Teachers’ Activities in the Process of Scientific Modelling.” Research in Science Education 49 (5): 1319–1345. doi:10.1007/s11165-017-9657-8.
   Web of Science ®Google Scholar
• Kurasaki, K. S. 2000. “Intercoder Reliability for Validating Conclusions Drawn from Open-ended Interview Data.” Field Methods 12: 179–194. doi:10.1177/1525822X0001200301.
   Google Scholar
• Lally, D., and C. Forbes. 2019. “Modelling Water Systems in an Introductory Undergraduate Course: Students’ Use and Evaluation of Data-driven, Computer-based Models.” International Journal of Science Education 41 (14): 1999–2023. doi:10.1080/09500693.2019.1657252.
   Web of Science ®Google Scholar
• Landis, J. R., and G. G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33 (1): 159–174. doi:10.2307/2529310.
   PubMed Web of Science ®Google Scholar
• Lane, A. K., C. Hardison, A. Simon, and T. C. Andrews. 2019. “A Model of the Factors Influencing Teaching Identity among Life Sciences Doctoral Students.” Journal of Research in Science Teaching 56 (2): 141–162. doi:10.1002/tea.21473.
   Web of Science ®Google Scholar
• Lavi, R., and Y. J. Dori. 2019. “Systems Thinking of Pre- and In-service Science and Engineering Teachers.” International Journal of Science Education 41 (2): 248–279. doi:10.1080/09500693.2018.1548788.
   Web of Science ®Google Scholar
• LeCompte, M. D., and J. P. Goetz. 1982. “Problems of Reliability and Validity in Ethnographic Research.” Review of Educational Research 52 (1): 31–60. doi:10.3102/00346543052001031.
   Web of Science ®Google Scholar
• *Lee, T. D., M. Gail Jones, and K. Chesnutt. 2019. “Teaching Systems Thinking in the Context of the Water Cycle.” Research in Science Education 49 (1): 137–172. doi:10.1007/s11165-017-9613-7.
   Web of Science ®Google Scholar
• *Liu, S., and G. Roehrig. 2019. “Exploring Science Teachers’ Argumentation and Personal Epistemology about Global Climate Change.” Research in Science Education 49 (1): 173–189. doi:10.1007/s11165-017-9617-3.
   Web of Science ®Google Scholar
• MacPhail, C., N. Khoza, L. Abler, and M. Ranganathan. 2016. “Process Guidelines for Establishing Intercoder Reliability in Qualitative Studies.” Qualitative Research 16 (2): 198–212. doi:10.1177/1468794115577012.
   Web of Science ®Google Scholar
• Marques, J. F., and C. McCall. 2005. “The Application of Interrater Reliability as a Solidification Instrument in a Phenomenological Study.” The Qualitative Report 10 (3): 439–462.
   Google Scholar
• *Martin, J. 2019. “Researching Primary Teachers’ Professional Agency: Employing Interactive Ethnography to Overcome Reluctance to Teach Science.” Research in Science Education 49 (5): 1279–1299. doi:10.1007/s11165-017-9654-y.
   Web of Science ®Google Scholar
• *Mathayas, N., D. E. Brown, R. C. Wallon, and R. Lindgren. 2019. “Representational Gesturing as an Epistemic Tool for the Development of Mechanistic Explanatory Models.” Science Education 103 (4): 1047–1079. doi:10.1002/sce.21516.
   Web of Science ®Google Scholar
• *Melville, W., T. Campbell, and D. Jones. 2019. “Axiology, the Subject and the Chair.” Research in Science Education 49 (3): 679–696. doi:10.1007/s11165-017-9646-y.
   Web of Science ®Google Scholar
• Morse, J. M., M. Barrett, M. Mayan, K. Olson, and J. Spiers. 2002. “Verification Strategies for Establishing Reliability and Validity in Qualitative Research.” International Journal of Qualitative Methods 1 (2): 1–19. doi:10.1177/160940690200100202.
   Google Scholar
• O’Connor, C., and H. Joffe. 2020. “Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines.” International Journal of Qualitative Methods 19: 1–13. doi:10.1177/1609406919899220.
   Web of Science ®Google Scholar
• *Overman, M., J. D. Vermunt, P. C. Meijer, and M. Brekelmans. 2019. “Teacher–student Negotiations during Context‐based Chemistry Reform: A Case Study.” Journal of Research in Science Teaching 56 (6): 797–820. doi:10.1002/tea.21528.
   Web of Science ®Google Scholar
• *Pattison, S. A., and L. D. Dierking. 2019. “Early Childhood Science Interest Development: Variation in Interest Patterns and Parent–child Interactions among Low‐income Families.” Science Education 103 (2): 362–388. doi:10.1002/sce.21486.
   Web of Science ®Google Scholar
• *Phillips, T. B., H. L. Ballard, B. V. Lewenstein, and R. Bonney. 2019. “Engagement in Science through Citizen Science: Moving beyond Data Collection.” Science Education 103 (3): 665–690. doi:10.1002/sce.21501.
   Web of Science ®Google Scholar
• Richards, K. 2003. Qualitative Research in TESOL. New York: Palgrave Macmillan.
   Google Scholar
• Ryan, G. W. 1999. “Measuring the Typicality of Text: Using Multiple Coders for More than Just Reliability and Validity Checks.” Human Organization 58: 312–322. doi:10.17730/humo.58.3.g224147522545rln.
   Web of Science ®Google Scholar
• *Schizas, D., E. Papatheodorou, and G. Stamou. 2019. “Unravelling the Holistic Nature of Ecosystems: Biology Teachers’ Conceptions of Ecosystem Balance and Self-regulation.” International Journal of Science Education 41 (18): 2626–2646. doi:10.1080/09500693.2019.1690179.
   Web of Science ®Google Scholar
• *Sheth, M. J. 2019. “Grappling with Racism as Foundational Practice of Science Teaching.” Science Education 103 (1): 37–60. doi:10.1002/sce.21450.
   Web of Science ®Google Scholar
• *Spektor-Levy, O., and M. Yifrach. 2019. “If Science Teachers Are Positively Inclined Toward Inclusive Education, Why Is It So Difficult?” Research in Science Education 49 (3): 737–766. doi:10.1007/s11165-017-9636-0.
   Web of Science ®Google Scholar
• Spradley, J. P. 1979. The Ethnographic Interview. New York: Holt: Rinehart and Winston.
   Google Scholar
• *Stokhof, H., B. De Vries, T. Bastiaens, and R. Martens. 2019. “Mind Map Our Way into Effective Student Questioning: A Principle-Based Scenario.” Research in Science Education 49 (2): 347–369. doi:10.1007/s11165-017-9625-3.
   Web of Science ®Google Scholar
• Sun, S. 2011. “Meta-analysis of Cohen’s Kappa.” Health Services & Outcomes Research Methodology 11 (3–4): 145–163. doi:10.1007/s10742-011-0077-3.
   Google Scholar
• Taber, K. S. 2013. Classroom-based Research and Evidence-based Practice: An Introduction. London: Sage.
   Google Scholar
• Taber, K. S. 2018. “The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education.” Research in Science Education 48 (6): 1273–1296. doi:10.1007/s11165-016-9602-2.
   Web of Science ®Google Scholar
• Tai, K. W. H. Forthcoming. “Translanguaging as Inclusive Pedagogical Practices in English Medium Instruction Science and Mathematics Classrooms for Linguistically and Culturally Diverse Students.” Research in Science Education. doi: 10.1007/s11165-021-10018-6
   Web of Science ®Google Scholar
• Tinsley, H. E., & D. J. Weisss. 2000. “Interrater reliability and agreement”. In Handbook of applied multivariate statistics and mathematical modeling (pp. 95–124). Academic Press.
   Google Scholar
• *Tsybulsky, D. 2019. “Students Meet Authentic Science: The Valence and Foci of Experiences Reported by High-school Biology Students regarding Their Participation in a Science Outreach Programme.” International Journal of Science Education 41 (5): 567–585. doi:10.1080/09500693.2019.1570380.
   Web of Science ®Google Scholar
• *Vo, T., C. Forbes, L. Zangori, and C. V. Schwarz. 2019. “Longitudinal Investigation of Primary Inservice Teachers’ Modelling the Hydrological Phenomena.” International Journal of Science Education 41 (18): 2788–2807. doi:10.1080/09500693.2019.1698786.
   Web of Science ®Google Scholar
• *Wade‐Jaimes, K., and R. Schwartz. 2019. ““I Don’t Think It’s Science:” African American Girls and the Figured World of School Science.” Journal of Research in Science Teaching 56 (6): 679–706. doi:10.1002/tea.21521.
   Web of Science ®Google Scholar
• *Zohar, A. R., and S. T. Levy. 2019. “Students’ Reasoning about Chemical Bonding: The Lacuna of Repulsion.” Journal of Research in Science Teaching 56 (7): 881–904. doi:10.1002/tea.21532.
   Web of Science ®Google Scholar