Review of measurements used in computing education research and suggestions for increasing standardization

References

• Ahadi, A., Lister, R., Haapala, H., & Vihavainen, A. (2015, July). Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research,Omaha, Nebraska, USA (pp. 121–130). ACM.
   Google Scholar
• Almstrum, V. L., Hazzan, O., Guzdial, M., & Petre, M. (2005). Challenges to computer science education research. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE ‘05) (pp. 191–192). New York, NY: ACM.
   Google Scholar
• Asterhan, C. S., & Schwarz, B. B. (2009). Argumentation and explanation in conceptual change: Indications from protocol analyses of peer-to‐peer dialog. Cognitive Science, 33(3), 374–400.
   PubMed Web of Science ®Google Scholar
• Atkins, S. M., Sprenger, A. M., Colflesh, G. J., Briner, T. L., Buchanan, J. B., Chavis, S. E., … Harbison, J. I. (2014). Measuring working memory is all fun and games. Experimental Psychology, 61,417–438.
   Google Scholar
• Basawapatna, A. R., Repenning, A., Koh, K. H., & Nickerson, H. (2013, August). The zones of proximal flow: Guiding students through a space of computational thinking skills and challenges. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, San Diego, California, USA (pp. 67–74). ACM.
   Google Scholar
• Bati, T. B., Gelderblom, H., & Van Biljon, J. (2014). A blended learning approach for teaching computer programming: Design for large classes in Sub-Saharan Africa. Computer Science Education, 24(1), 71–99.
   Google Scholar
• Behnke, K. A., Kos, B. A., & Bennett, J. K. (2016, August). Computer science principles: Impacting student motivation & learning within and beyond the classroom. In Proceedings of the 2016 ACM Conference on International Computing Education Research , Melbourne, Australia (pp. 171–180). ACM.
   Google Scholar
• Belbin, R. M. (1993). Team roles at work. Oxford, UK: Butterworth-Heinemann.
   Google Scholar
• Ben-David Kolikant, Y., & Genut, S. (2017). The effect of prior education on students’ competency in digital logic: The case of ultraorthodox Jewish students. Computer Science Education, 27(3–4), 149–174.
   Web of Science ®Google Scholar
• Beyer, S. (2014). Why are women underrepresented in computer science? Gender differences in stereotypes, self-efficacy, values, and interests and predictors of future CS course-taking and grades. Computer Science Education, 24(2–3), 153–192.
   Google Scholar
• Bhardwaj, J. (2017). In search of self-efficacy: Development of a new instrument for first year Computer Science students. Computer Science Education, 27(2), 79–99.
   Web of Science ®Google Scholar
• Biggs, J., Kember, D., & Leung, D. Y. (2001). The revised two-factor study process questionnaire: R-SPQ-2F. British Journal of Educational Psychology, 71, 133–149.
   PubMed Web of Science ®Google Scholar
• Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246–263.
   PubMed Web of Science ®Google Scholar
• Brown, N. C., & Altadmri, A. (2014, July). Investigating novice programming mistakes: Educator beliefs vs. student data. In Proceedings of the Tenth Annual Conference on International Computing Education Research, Glasgow, Scotland, UK (pp. 43–50). ACM.
   Google Scholar
• Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015, July). The normalized programming state model: Predicting student performance in computing courses based on programming behavior. Proceedings of the Eleventh Annual International Conference on International Computing Education Research,Omaha, Nebraska, USA (pp. 141–150). ACM.
   Google Scholar
• Cetin, I. (2013). Visualization: A tool for enhancing students’ concept images of basic object-oriented concepts. Computer Science Education, 23(1), 1–23.
   Google Scholar
• Cetin, I., & Ozden, M. Y. (2015). Development of computer programming attitude scale for university students. Computer Applications in Engineering Education, 23(5), 667–672.
   Web of Science ®Google Scholar
• Cooper, S., Wang, K., Israni, M., & Sorby, S. (2015, July). Spatial skills training in introductory computing. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research,Omaha, Nebraska, USA (pp. 13–20). ACM.
   Google Scholar
• Cross, S. E., Bacon, P. L., & Morris, M. L. (2000). The relational-interdependent self-construal and relationships. Journal of Personality and Social Psychology, 78, 791–808.
   PubMed Web of Science ®Google Scholar
• Danielsiek, H., Toma, L., & Vahrenhold, J. (2017, August). An instrument to assess self-efficacy in introductory algorithms courses. In Proceedings of the 2017 ACM Conference on International Computing Education Research,Melbourne, Australia (pp. 217–225). ACM.
   Google Scholar
• Deci, E. L., & Ryan, R. M. (2003). Intrinsic motivation inventory. Self-determination theory, 267. Retrieved from http://selfdeterminationtheory.org/questionnaires/10-questionnaires/50
   Google Scholar
• Decker, A., & McGill, M. M. (2017, March 8–11). Pre-college computing outreach research: Towards improving the practice. In Proceedings of the 48th SIGCSE Technical Symposium of Computer Science Education (pp. 153–158). Seattle, WA: ACM.
   Google Scholar
• Decker, A., McGill, M. M., & Settle, A. (2016, March 2–5). Towards a common framework for evaluating computing outreach activities. In Proceedings of the 47th SIGCSE Technical Symposium of Computer Science Education (pp. 627–632). Memphis, TN: ACM.
   Google Scholar
• Dorn, B., & Elliott Tew, A. (2015). Empirical validation and application of the computing attitudes survey. Computer Science Education, 25(1), 1–36.
   Web of Science ®Google Scholar
• Eagan, B. R., Rogers, B., Serlin, R., Ruis, A. R., Arastoopour Irgens, G., & Shaffer, D. W. (2017). Can we rely on IRR? Testing the assumptions of inter-rater reliability. In International Conference on Computer Supported Collaborative Learning,Philadelphia, PA, USA.
   Google Scholar
• Ellitot Tew, A., & Guzdial, M. (2010, March). Developing a validated assessment of fundamental CS1 concepts. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education,Milwaukee, WI (pp. 97–101). ACM.
   Google Scholar
• Every Student Succeeds Act. (2015). S.1177—114thCongress. Retrieved from https://www.govinfo.gov/app/details/CRPT-114hrpt354/CRPT-114hrpt354/context
   Google Scholar
• Fisler, K. (2014, July). The recurring rainfall problem. Proceedings of the Tenth Annual Conference on International Computing Education Research, Glasgow, Scotland, UK (pp. 35–42). ACM.
   Google Scholar
• Flanigan, A. E., Peteranetz, M. S., Shell, D. F., & Soh, L. K. (2015, July). Exploring changes in computer science students’ implicit theories of intelligence across the semester. Proceedings of the eleventh annual International Conference on International Computing Education Research,Omaha, NE, USA (pp. 161–168). ACM.
   Google Scholar
• Fronza, I., Ioini, N. E., & Corral, L. (2017). Teaching computational thinking using agile software engineering methods: A framework for middle schools. ACM Transactions on Computing Education (TOCE), 17(4), 19.
   Web of Science ®Google Scholar
• Hamouda, S., Edwards, S. H., Elmongui, H. G., Ernst, J. V., & Shaffer, C. A. (2017). A basic recursion concept inventory. Computer Science Education, 27(2), 121–148.
   Web of Science ®Google Scholar
• Harms, K. J., Chen, J., & Kelleher, C. L. (2016, August). Distractors in Parsons problems decrease learning efficiency for young novice programmers. In Proceedings of the 2016 ACM Conference on International Computing Education Research,Melbourne, Australia (pp. 241–250). ACM.
   Google Scholar
• Herman, G. L., Zilles, C., & Loui, M. C. (2014). A psychometric evaluation of the digital logic concept inventory. Computer Science Education, 24(4), 277–303.
   Google Scholar
• Hristova, M., Misra, A., Rutter, M., & Mercuri, R. (2003). Identifying and correcting Java programming errors for introductory computer science students. In ACM SIGCSE Bulletin (35 (1), pp. 153-156). ACM: New York, NY..
   Google Scholar
• Hundhausen, C. D., Agrawal, A., & Agarwal, P. (2013). Talking about code: Integrating pedagogical code reviews into early computing courses. ACM Transactions on Computing Education (TOCE), 13(3), 14.
   Web of Science ®Google Scholar
• Husman, J., Derryberry, W. P., Crowson, H. M., & Lomax, R. (2004). Instrumentality, task value, and intrinsic motivation: Making sense of their independent interdependence. Contemporary Educational Psychology, 29(1), 63–76.
   Web of Science ®Google Scholar
• Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., & Rubio, M. Á. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. In Proceedings of the 2015 ITiCSE on Working Group Reports (pp. 41-63). ACM.
   Google Scholar
• Israel, M., Wherfel, Q. M., Shehab, S., Melvin, O., & Lash, T. (2017, August). Describing elementary students’ interactions in K-5 puzzle-based computer science environments using the Collaborative Computing Observation Instrument (C-COI). Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, WA, USA (pp. 110–117). ACM.
   Google Scholar
• Israel, M., Wherfel, Q. M., Shehab, S., Ramos, E. A., Metzger, A., & Reese, G. C. (2016). Assessing collaborative computing: Development of the collaborative-computing observation instrument (C-COI). Computer Science Education, 26(2–3), 208–233.
   Web of Science ®Google Scholar
• Jadud, M. C., & Dorn, B. (2015, July). Aggregate compilation behavior: Findings and implications from 27,698 users. Proceedings of the Eleventh Annual International Conference on International Computing Education Research, Omaha, NE, USA (pp. 131–139). ACM.
   Google Scholar
• John, O. P., & Srivastava, S. (1999). The big five trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin & O. P. John (Eds.), Handbook of personality: Theory and research (2nd ed., pp. 102–138). New York, NY: Guilford Press.
   Google Scholar
• Johns, M. W. (1991). A new method for measuring daytime sleepiness: The Epworth sleepiness scale. Sleep, 14(6), 540–545.
   PubMed Web of Science ®Google Scholar
• Khan, K. S., Kunz, R., Kleijnen, J., & Antes, G. (2003). Five steps to conducting a systematic review. Journal of the Royal Society of Medicine (JRSM), 96(3), 118–121.
   PubMed Web of Science ®Google Scholar
• Ko, A. J., & Davis, K. (2017, August). Computing mentorship in a software boomtown: Relationships to adolescent interest and beliefs. Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, WA, USA (pp. 236–244). ACM.
   Google Scholar
• Koh, K. H., Nickerson, H., Basawapatna, A., & Repenning, A. (2014, June). Early validation of computational thinking pattern analysis. Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education, Uppsala, Sweden (pp. 213–218). ACM.
   Google Scholar
• Kurkovsky, S. (2013). Mobile game development: Improving student engagement and motivation in introductory computing courses. Computer Science Education, 23(2), 138–157.
   Google Scholar
• Lee, C. B., Garcia, S., & Porter, L. (2013). Can peer instruction be effective in upper-division computer science courses? ACM Transactions on Computing Education (TOCE), 13(3), 12.
   Web of Science ®Google Scholar
• Lee, M. J., & Ko, A. J. (2015, July). Comparing the effectiveness of online learning approaches on CS1 learning outcomes. Proceedings of the Eleventh Annual International Conference on International Computing Education Research, Omaha, NE, USA (pp. 237–246). ACM.
   Google Scholar
• Leppink, J., Paas, F., Van der Vleuten, C. P., Van Gog, T., & Van Merriënboer, J. J. (2013). Development of an instrument for measuring different types of cognitive load. Behavior Research Methods, 45(4), 1058–1072.
   PubMed Web of Science ®Google Scholar
• Lewis, C. M., Khayrallah, H., & Tsai, A. (2013, August). Mining data from the AP CS A exam: Patterns, non-patterns, and replication failure. Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, San Diego, CA, USA (pp. 115–122). ACM.
   Google Scholar
• Lips, H. M. (1992). Gender- and science-related attitudes as predictors of college students’ academic choices. Journal of Vocational Behavior, 40, 62–81.
   Web of Science ®Google Scholar
• Lishinski, A., Good, J., Sands, P., & Yadav, A. (2016, August). Methodological rigor and theoretical foundations of CS education research. Proceedings of the 2016 ACM Conference on International Computing Education Research, Melbourne, Australia (pp. 161–169). ACM.
   Google Scholar
• Magerko, B., Freeman, J., Mcklin, T., Reilly, M., Livingston, E., Mccoid, S., & Crews-Brown, A. (2016). Earsketch: A steam-based approach for underrepresented populations in high school computer science education. ACM Transactions on Computing Education (TOCE), 16(4), 14.
   Web of Science ®Google Scholar
• Margolis, J., Estella, R., Goode, J., Holme, J., & Nao, K. (2008). Stuck in the shallow end: Education, race, and computing. Cambridge, MA: MIT Press.
   Google Scholar
• Margulieux, L., & Catrambone, R. (2017, August). Using learners’ self-explanations of subgoals to guide initial problem solving in app inventor. In Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, WA, USA (pp. 21–29). ACM.
   Google Scholar
• Marshall, L., Pieterse, V., Thompson, L., & Venter, D. M. (2016). Exploration of participation in student software engineering teams. ACM Transactions on Computing Education (TOCE), 16(2), 5.
   Web of Science ®Google Scholar
• Martin, J., Edwards, S. H., & Shaffer, C. A. (2015, July). The effects of procrastination interventions on programming project success. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research ,Omaha, NE, USA (pp. 3–11). ACM.
   Google Scholar
• McCabe, T. J. (1976). A complexity measure. IEEE Transactions on Software Engineering, (4), 308–320.
   Google Scholar
• McGill, M. M., Decker, A., & Abbott, Z. (2018, February). Improving research and experience reports of pre-college computing activities: A gap analysis. Proceedings of the 49th ACM Technical Symposium on Computer Science Education, Baltimore, Maryland, USA (pp. 964–969). ACM.
   Google Scholar
• Morrison, B. B. (2017, August). Dual modality code explanations for novices: Unexpected results. Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, WA, USA (pp. 226–235). ACM.
   Google Scholar
• Morrison, B. B., Dorn, B., & Guzdial, M. (2014, July). Measuring cognitive load in introductory CS: Adaptation of an instrument. Proceedings of the 10th Annual Conference on ICER, Glasgow, Scotland, UK (pp. 131–138). ACM.
   Google Scholar
• Morrison, B. B., Margulieux, L. E., & Guzdial, M. (2015, July). Subgoals, context, and worked examples in learning computing problem solving. Proceedings of the Eleventh Annual International Conference on International Computing Education Research, Omaha, NE, USA (pp. 21–29). ACM.
   Google Scholar
• Myers, I. B., McCaulley, M. H., & Most, R. (1985). Manual: A guide to the development and use of the Myers-Briggs type indicator. Palo Alto, CA: Consulting Psychologists Press.
   Google Scholar
• Nathan, M. J., & Sawyer, R. K. (2014). Foundations of the learning sciences. In the Cambridge handbook of the learning sciences (2nd ed., pp. 21–43). Cambridge, UK: Cambridge University Press.
   Google Scholar
• National Board for Education Sciences. (2015). National board for education sciences: 2014 NBES annual report, July 2013 through June 2014. Washington, DC: US Department of Education.
   Google Scholar
• Nelson, G. L., Xie, B., & Ko, A. J. (2017, August). Comprehension first: Evaluating a novel pedagogy and tutoring system for program tracing in CS1. In Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, WA, USA (pp. 2–11). ACM.
   Google Scholar
• Oh, Y. J., Jia, Y., Lorentson, M., & LaBanca, F. (2013). Development of the educational and career interest scale in science, technology, and mathematics for high school students. Journal of Science Education and Technology, 22(5), 780–790.
   Web of Science ®Google Scholar
• Öhrstedt, M. (2009). Approaches to studying, stress, academic achievement and the ability to assess own performance [Studieapproach, stress, studieresultat och förmåga att bedöma egen prestation] (Master’s thesis). Stockholm: Department of psychology, Stockholm university.
   Google Scholar
• Parker, M. C., Guzdial, M., & Engleman, S. (2016, August). Replication, validation, and use of a language independent CS1 knowledge assessment. Proceedings of the 2016 Conference on ICER, Melbourne, Australia (pp. 93–101). ACM.
   Google Scholar
• Pekrun, R., Vom Hofe, R., Blum, W., Frenzel, A. C., Götz, T., & Wartha, S. (2007). Development of mathematical competencies in adolescence: The PALMA longitudinal study. Munster, Germany: Waxmann.
   Google Scholar
• Peteranetz, M. S., Flanigan, A. E., Shell, D. F., & Soh, L. K. (2016, August). Perceived instrumentality and career aspirations in CS1 courses: Change and relationships with achievement. Proceedings of the 2016 ACM Conference on International Computing Education Research, Melbourne, Australia (pp. 13–21). ACM.
   Google Scholar
• Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Oxford: Blackwell Publishing.
   Google Scholar
• Pintrich, P. R., Smith, D. A., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813.
   Web of Science ®Google Scholar
• Porter, L., Zingaro, D., & Lister, R. (2014, July). Predicting student success using fine grain clicker data. Proceedings of the Tenth Annual Conference on International Computing Education Research, Glasgow, Scotland, UK (pp. 51–58). ACM.
   Google Scholar
• Ramalingam, V., & Wiedenbeck, S. (1998). Development and validation of scores on a computer programming self-efficacy scale and group analyses of novice programmer self-efficacy. Journal of Educational Computing Research, 19(4), 367–381.
   Web of Science ®Google Scholar
• Repenning, A., Webb, D. C., Koh, K. H., Nickerson, H., Miller, S. B., Brand, C., … Repenning, N. (2015). Scalable game design: A strategy to bring systemic computer science education to schools through game design and simulation creation. ACM Transactions on Computing Education (TOCE), 15(2), 11.
   Web of Science ®Google Scholar
• Robins, A. (2015). The ongoing challenges of computer science education research. Computer Science Education, 25(2), 115–119.
   Web of Science ®Google Scholar
• Rosen, J. A., Porter, S. R., & Rogers, J. (2017). Understanding student self-reports of academic performance and course-taking behavior. AERA Open, 3(2), 1–14.
   Google Scholar
• Rountree, N., Rountree, J., Robins, A., & Hannah, R. (2004). Interacting factors that predict success and failure in a CSI course. SIGCSE Bulletin, 33(4), 101–104.
   Google Scholar
• Schneider, K., Liskin, O., Paulsen, H., & Kauffeld, S. (2015). Media, mood, and meetings: Related to project success? ACM Transactions on Computing Education (TOCE), 15(4), 21.
   Web of Science ®Google Scholar
• Scott, M. J., & Ghinea, G. (2014, July). Measuring enrichment: The assembly and validation of an instrument to assess student self-beliefs in CS1. Proceedings of the Tenth Annual Conference on International Computing Education Research, Glasgow, Scotland, UK (pp. 123–130). ACM.
   Google Scholar
• Shell, D. F., & Husman, J. (2008). Control, motivation, affect, and strategic self-regulation in the college classroom: A multidimensional phenomenon. Journal of Educational Psychology, 100(2), 443.
   Web of Science ®Google Scholar
• Simon, B., Esper, S., Porter, L., & Cutts, Q. (2013, August). Student experience in a student-centered peer instruction classroom. Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, San Diego, CA, USA (pp. 129–136). ACM.
   Google Scholar
• Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., & Su, T. T. (2009). Why peer discussion improves student performance on in-class concept questions. Science, 323(5910), 122–124.
   PubMed Web of Science ®Google Scholar
• Snow, E., Rutstein, D., Bienkowski, M., & Xu, Y. (2017, August). Principled assessment of student learning in high school computer science. Proceedings of the 2017 ACM Conference on International Computing Education Research, Tacoma, WA, USA (pp. 209–216). ACM.
   Google Scholar
• Sommerhoff, D., Szameitat, A., Vogel, F., Chernikova, O., Loderer, K., & Fischer, F. (2018). What do we teach when we teach the learning sciences? A document analysis of 75 graduate programs. Journal of the Learning Sciences 27(2), 319-351.
   Web of Science ®Google Scholar
• Stout, J. G., & Blaney, J. M. (2017). “But it doesn’t come naturally”: How effort expenditure shapes the benefit of growth mindset on women’s sense of intellectual belonging in computing. Computer Science Education, 27(3–4), 215–228.
   Web of Science ®Google Scholar
• Svedin, M., & Bälter, O. (2016). Gender neutrality improved completion rate for all. Computer Science Education, 26(2–3), 192–207.
   Web of Science ®Google Scholar
• Theodoropoulos, A., Antoniou, A., & Lepouras, G. (2017). How do different cognitive styles affect learning programming? Insights from a game-based approach in Greek schools. ACM Transactions on Computing Education (TOCE), 17(1), 3.
   Web of Science ®Google Scholar
• Trochim, W. M. (2006). The research methods knowledge base (2nd ed.). Retrieved from http://www.socialresearchmethods.net/kb/.
   Google Scholar
• Tuckman, B. W. (1991). The development and concurrent validity of the procrastination scale. Educational and Psychological Measurement, 51(2), 473–480.
   Web of Science ®Google Scholar
• Vahrenhold, J., & Paul, W. (2014). Developing and validating test items for first-year computer science courses. Computer Science Education, 24(4), 304–333.
   Google Scholar
• Vihavainen, A., Airaksinen, J., & Watson, C. (2014, July). A systematic review of approaches for teaching introductory programming and their influence on success. In Proceedings of the Tenth Annual Conference on International Computing Education Research, Glasgow, Scotland, UK (pp. 19–26). ACM.
   Google Scholar
• Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063.
   PubMed Web of Science ®Google Scholar
• Yoon, S. Y. (2011). Psychometric properties of the revised Purdue spatial visualization tests: Visualization of rotations (The revised PSVT: R). West Lafayette, Indiana: Purdue University.
   Google Scholar