Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes

References

• Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college. Washington, DC: US Department of Education. Retrieved from: http://files.eric.ed.gov/fulltext/ED490195.pdf.
   Google Scholar
• Agasisti, T., & Bowers, A. J. (2017). Data analytics and decision-making in education: Towards the educational data scientist as a key actor in schools and higher education institutions. In G. Johnes, J. Johnes, T. Agasisti, L. López-Torres (Eds.), Handbook of contemporary education economics (pp. 184–210). Cheltenham, UK: Edward Elgar Publishing. https://doi.org/10.7916/D8PR95T2.
   Google Scholar
• Allensworth, E. M. (2006). From high school to the future: A first look at Chicago public school graduates' college enrollment, college preparation, and graduation from four-year colleges. Chicago, IL: Consortium on Chicago School Research. http://files.eric.ed.gov/fulltext/ED499368.pdf
   Google Scholar
• Allensworth, E. M. (2013). The use of ninth-grade early warning indicators to improve Chicago schools. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 68–83. doi: 10.1080/10824669.2013.745181
   Google Scholar
• Allensworth, E. M., & Easton, J. Q. (2007). What matters for staying on-track and graduating in Chicago public highs schools: A close look at course grades, failures, and attendance in the freshman year. Research report. Chicago, IL: Consortium on Chicago School Research, http://files.eric.ed.gov/fulltext/ED498350.pdf.
   Google Scholar
• Allensworth, E. M., Gwynne, J. A., Moore, P., & de la Torre, M. (2014). Looking forward to high school and college: Middle grade indicators of readiness in Chicago public schools. Chicago, IL: University of Chicago Consortium on Chicago School Research. http://files.eric.ed.gov/fulltext/ED553149.pdf
   Google Scholar
• Allensworth, E., & Luppescu, S. (2018). Why do students get good grades, or bad ones? The influence of the teacher, class, school, and student. Chicago, IL: Retrieved from https://consortium.uchicago.edu/sites/default/files/publications/Why%20Do%20Students%20Get-Apr2018-Consortium.pdf.
   Google Scholar
• Allensworth, E. M., Nagaoka, J., & Johnson, D. W. (2018). High school graduation and college readiness indicator systems: What we know, what we need to know. Chicago, IL: University of Chicago Consortium on School Research.
   Google Scholar
• Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12(4), 387–415. doi: 10.1016/0022-2496(75)90001-2
   Web of Science ®Google Scholar
• Baker, R. S. (2013). Learning, schooling, and data analytics. In M. Murphy, S. Redding, & J. Twyman (Eds.), Handbook on innovations in learning. Philadelphia, PA: Center on Innovations in Learning, Temple University; Charlotte, NC: Information Age Publishing. Retrieved from http://www.centeril.org/.
   Google Scholar
• Baker, R. S. (2014). Educational data mining: An advance for intelligent systems in education. IEEE Intelligent Systems, 29(3), 78–82. doi: 10.1109/MIS.2014.42
   Web of Science ®Google Scholar
• Baker, R. S. (2015). Big data and education (2nd ed.). New York, NY: Teachers College, Columbia University.
   Google Scholar
• Balfanz, R., & Boccanfuso, C. (2007). Falling off the path to graduation: Middle grade indicators in [an unidentified northeastern city]. Baltimore, MD: Center for Social Organization of Schools.
   Google Scholar
• Balfanz, R., Herzog, L., & Mac Iver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist, 42(4), 223–235. http://www.tandfonline.com/doi/pdf/10.1080/00461520701621079?needAccess=true. doi: 10.1080/00461520701621079
   Web of Science ®Google Scholar
• Bandeira de Mello, V., Bohrnstedt, G., Blankenship, C., & Sherman, D. (2015). Mapping state proficiency standards onto NAEP scales: Results from the 2013 NAEP reading and mathematics assessments (NCES 2015-046) (NCES 2015-046). Washington, DC. Retrieved from https://files.eric.ed.gov/fulltext/ED557749.pdf.
   Google Scholar
• Becker, J., Schools, P. P., Levinger, B., Schools, P. G. S. C. P., Sims, A., & Whittington, A. (2014). Student success and college readiness: Translating predictive analytics into action. http://sdp.cepr.harvard.edu/files/cepr-sdp/files/sdp-fellowship-capstone-student-success-college-readiness.pdf
   Google Scholar
• Bowen, W. G., Chingos, M. M., & McPherson, M. S. (2009). Crossing the finish line: Completing college at America's public universities. Princeton, NJ: Princeton University Press.
   Google Scholar
• Bowers, A. J. (in press). Report card grades and educational outcomes. In S. M. Brookhart & T. R. Guskey (Eds.), What do we know about grading? Alexandria, VA: Association for Supervision and Curriculum Development (ASCD).
   Google Scholar
• Bowers, A. J. (2017). Quantitative research methods training in education leadership and administration preparation programs as disciplined inquiry for building school improvement capacity. Journal of Research on Leadership Education, 12(1), 72–96. doi: 10.1177/194277511665946
   Web of Science ®Google Scholar
• Bowers, A. J. (2011). What's in a grade? The multidimensional nature of what teacher-assigned grades assess in high school. Educational Research and Evaluation, 17(3), 141–159. doi: 10.1080/13803611.2011.597112
   Google Scholar
• Bowers, A. J. (2010). Analyzing the longitudinal K-12 grading histories of entire cohorts of students: Grades, data driven decision making, dropping out and hierarchical cluster analysis. Practical Assessment Research and Evaluation, 15(7), 1–18. http://www.pareonline.net/pdf/v15n7.pdf
   Google Scholar
• Bowers, A. J. (2009). Reconsidering grades as data for decision making: More than just academic knowledge. Journal of Educational Administration, 47(5), 609–629. doi: 10.1108/09578230910981080
   Google Scholar
• Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. https://muse.jhu.edu/article/496998/pdf
   Google Scholar
• Bowers, A. J., & Sprott, R. (2012a). Why tenth graders fail to finish high school: A dropout typology latent class analysis. The Journal of Education for Students Placed at Risk (JESPAR), 17(3), 129–148. doi: 10.1080/10824669.2012.692071
   Google Scholar
• Bowers, A. J., & Sprott, R. (2012b). Examining the multiple trajectories associated with dropping out of high school: A growth mixture model analysis. The Journal of Educational Research, 105(3), 176–195. doi: 10.1080/00220671.2011.552075
   Web of Science ®Google Scholar
• Brookhart, S. M., Guskey, T. R., Bowers, A. J., McMillan, J. H., Smith, J. K., Smith, L. F., … Welsh, M. E. (2016). A century of grading research: Meaning and value in the most common educational measure. Review of Educational Research, 86(4), 803–848. doi: 10.3102/0034654316672069
   Web of Science ®Google Scholar
• Blickenstaff, C. (2005). Women and science careers: leaky pipeline or gender filter? Gender and Education, 17(4), 369–386. doi: 10.1080/09540250500145072
   Web of Science ®Google Scholar
• Carlson, S. E. (2018). Identifying students at risk of dropping out: indicators and thresholds using ROC analysis. http://digitalcommons.georgefox.edu/edd/114
   Google Scholar
• Chen, X. (2013). STEM Attrition: College students' paths into and out of STEM fields. Statistical Analysis Report. NCES 2014-001. Washington, DC: National Center for Education Statistics.
   Google Scholar
• Crisp, G., Nora, A., & Taggart, A. (2009). Student characteristics, pre-college, college, and environmental factors as predictors of majoring in and earning a STEM degree: An analysis of students attending a Hispanic serving institution. American Educational Research Journal, 46(4), 924–942. doi: 10.3102/0002831209349460
   Web of Science ®Google Scholar
• Cummings, K. D., & Smolkowski, K. (2015). Selecting students at risk of academic difficulties. Assessment for Effective Intervention, 41(1), 55–61. doi: 10.1177/1534508415590396
   Web of Science ®Google Scholar
• D'Agostino, J. V., Rodgers, E., & Mauck, S. (2018). Addressing inadequacies of the observation survey of early literacy achievement. Reading Research Quarterly, 53(1), 51–69. doi: 10.1002/rrq.181
   Web of Science ®Google Scholar
• Davis, M., Herzog, L., & Legters, N. (2013). Organizing schools to address Early Warning Indicators (EWIs): Common practices and challenges. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 84–100. doi: 10.1080/10824669.2013.745210
   Google Scholar
• DeLong, E., DeLong, D., & Clarke-Pearson, D. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845. doi: 10.2307/2531595. http://www.jstor.org/stable/2531595
   PubMed Web of Science ®Google Scholar
• Dutta-Moscato, J., Gopalakrishnan, V., Lotze, M. T., & Becich, M. J. (2014). Creating a pipeline of talent for informatics: STEM initiative for high school students in computer science, biology, and biomedical informatics. Journal of Pathology Informatics, 5(1), 12. doi: 10.4103/2153-3539.129448
   PubMedGoogle Scholar
• Engberg, M. E., & Wolniak, G. C. (2013). College student pathways to the STEM disciplines. Teachers College Record, 115(1), 1–27.
   Web of Science ®Google Scholar
• Fanjul-Hevia, A., & González-Manteiga, W. (2018). A comparative study of methods for testing the equality of two or more ROC curves. Computational Statistics, 1, 21. doi: 10.1007/s00180-017-0783-6
   Google Scholar
• Faria, A. M., Sorensen, N., Heppen, J., Bowdon, J., Taylor, S., Eisner, R., & Foster, S. (2017). Getting students on track for graduation: Impacts of the early warning intervention and monitoring system after one year. REL 2017-272. Regional Educational Laboratory Midwest, https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=388.
   Google Scholar
• Fogarty, J., Baker, R. S., & Hudson, S. E. (2005). Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction. In Proceedings of Graphics Interface 2005 (pp. 129–136). Victoria, BC: Canadian Human-Computer Communications Society.
   Google Scholar
• Frazelle, S., & Nagel, A. (2015). A practitioner's guide to implementing early warning systems. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest.
   Google Scholar
• Geiser, S., & Santelices, M. V. (2007). Validity of high-school grades in predicting student success beyond the freshman year: High-school record vs. standardized tests as indicators of four-year college outcomes. Research & Occasional Paper Series: CSHE. 6.07. Center for Studies in Higher Education, http://files.eric.ed.gov/fulltext/ED502858.pdf
   Google Scholar
• Gleason, P., & Dynarski, M. (2002). Do we know whom to serve? Issues in using risk factors to identify dropouts. Journal of Education for Students Students Placed at Risk, 7(1), 25–41. doi: 10.1207/S15327671ESPR0701_3
   Google Scholar
• Gönen, M. (2007). Analyzing receiver operating characteristic curves with SAS. Cary, NC: SAS Publishing.
   Google Scholar
• Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. doi: 10.1148/radiology.143.1.7063747
   PubMed Web of Science ®Google Scholar
• Hanley, J. A., & McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3), 839–843. doi: 10.1148/radiology.148.3.6878708.
   PubMed Web of Science ®Google Scholar
• Horn, A. S., & Lee, G. (2017). Evaluating the accuracy of productivity indicators in performance funding models. Educational Policy, 1–32. doi: 10.1177/0895904817719521
   Google Scholar
• Hughes, J., & Petscher, Y. (2016). A guide to developing and evaluating a college readiness screener. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast.
   Google Scholar
• Ikbal, S., Tamhane, A., Sengupta, B., Chetlur, M., Ghosh, S., & Appleton, J. (2015). On early prediction of risks in academic performance for students. IBM Journal of Research and Development, 59(6), 5:1–5:14. doi: 10.1147/JRD.2015.2458631
   Web of Science ®Google Scholar
• Ingels, S. J., Pratt, D. J., Alexander, C. P., Jewell, D. M., Lauff, E., Mattox, T. L., … Christopher, E. (2014). Education Longitudinal Study of 2002 (ELS:2002) Third Follow-Up Data File Documentation (NCES 2014-364). Washington, DC. Retrieved from: http://nces.ed.gov/pubs2014/2014364.pdf
   Google Scholar
• Jeni, L. A., Cohn, J. F., & De La Torre, F. (2013). Facing imbalanced data–recommendations for the use of performance metrics. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on (pp. 245–251) Geneva, Switzerland. IEEE. DOI:10.1109/ACII.2013.47
   Google Scholar
• Johnson, E., & Semmelroth, C. (2010). The predictive validity of the early warning system tool. NASSP Bulletin, 94(2), 120–134. doi: 10.1177/0192636510380789
   Google Scholar
• Jordan, N., Glutting, J., Ramineni, C., & Watkins, M. (2010). Validating a number sense screening tool for use in kindergarten and first grade: Prediction of mathematics proficiency in third grade. School Psychology Review, 39(2), 181–195. Retrieved from http://ezproxy.cul.columbia.edu/login?url= https://search.proquest.com/docview/608709943?accountid=10226
   Web of Science ®Google Scholar
• Kemple, J. J., Segeritz, M. D., & Stephenson, N. (2013). Building on-track indicators for high school graduation and college readiness: Evidence from New York City. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 7–28. doi: 10.1080/10824669.2013.747945
   Google Scholar
• Knight, S., & Shum, S. B. (2018). Theory and learning analytics. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of learning analytics (pp. 17–22). Tempe, AZ: Society for Learning Analytics Research.
   Google Scholar
• Knowles, J. E. (2015). Of needles and haystacks: Building an accurate statewide dropout early warning system in Wisconsin. JEDM-Journal of Educational Data Mining, 7(3), 18–67. Retrieved from https://jedm.educationaldatamining.org/index.php/JEDM/article/view/JEDM082.
   Google Scholar
• Koon, S., & Petscher, Y. (2016). Can scores on an interim high school reading assessment accurately predict low performance on college readiness exams? Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast.
   Google Scholar
• Krumm, A. E., Means, B., & Bienkowski, M. (2018). Learning analytics goes to school: A collaborative approach to improving education. New York, NY: Routledge.
   Google Scholar
• Lacefield, W. E., & Applegate, E. B. (2018). Data visualization in public education: Longitudinal student-, intervention-, school-, and district-level performance modeling. Paper presented at the Annual meeting of the American Educational Research Association, New York, NY.
   Google Scholar
• Lacefield, W., Applegate, B., Zeller, P. J., & Carpenter, S. (2012). Tracking students' academic progress in data rich but analytically poor environments. Paper presented at the Annual meeting of the American Educational Research Association, Vancouver, BC.
   Google Scholar
• Langenbucher, J., Labouvie, E., & Morgenstern, J. (1996). Measuring diagnostic agreement. Journal of Consulting and Clinical Psychology, 64(6), 1285. doi: 10.1037/0022-006X.64.6.1285
   PubMed Web of Science ®Google Scholar
• Laracy, S. D., Hojnoski, R. L., & Dever, B. V. (2016). Assessing the classification accuracy of early numeracy curriculum-based measures using receiver operating characteristic curve analysis. Assessment for Effective Intervention, 41(3), 172–183. doi: 10.1177/1534508415621542
   Web of Science ®Google Scholar
• Liao, H. F., Yao, G., Chien, C. C., Cheng, L. Y., & Hsieh, W. S. (2014). Likelihood ratios of multiple cutoff points of the Taipei City Developmental Checklist for Preschoolers, 2nd version. Journal of the Formosan Medical Association, 113(3), 179–186. doi: 10.1016/j.jfma.2011.10.55
   PubMed Web of Science ®Google Scholar
• López-Ratón, M., Rodríguez-Álvarez, M. X., Suarez, C. C., & Sampedro, F. G. (2014). OptimalCutpoints: an R package for selecting optimal cutpoints in diagnostic tests. Journal of Statistical Software, 61(8), 1–36.
   Web of Science ®Google Scholar
• Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi: 10.1016/j.compedu.2009.09.008
   Web of Science ®Google Scholar
• Mahoney, J. L., & Cairns, R. B. (1997). Do extracurricular activities protect against early school dropout? Developmental Psychology, 33(2), 241. doi: 10.1037/0012-1649.33.2.241
   PubMed Web of Science ®Google Scholar
• National Center on Response to Intervention. (2010). Users guide to universal screening tools chart. Washington, DC: National Center on Response to Intervention, Office of Special Education Programs, U.S. Department of Education. Retrieved from http://www.rti4success.org/sites/default/files/UniversalScreeningUsersGuide.pdf
   Google Scholar
• Neild, R. C., Balfanz, R., & Herzog, L. (2007). An early warning system. Educational Leadership, 65(2), 28–33. Retrieved from http://new.every1graduates.org/wp-content/uploads/2012/03/Early_Warning_System_Neild_Balfanz_Herzog.Pdf
   Web of Science ®Google Scholar
• Nicholls, G., Wolfe, H., Besterfield-Sacre, M., & Shuman, L. (2010). Predicting STEM degree outcomes based on eighth grade data and standard test scores. Journal of Engineering Education, 99(3), 209–223. doi: 10.1002/j.2168-9830.2010.tb01057.x
   Web of Science ®Google Scholar
• Noble, J. P., & Sawyer, R. L. (2004). Is high school GPA better than admission test scores for predicting academic success in college? College and University, 79(4), 17–22.
   Google Scholar
• Ocumpaugh, J., Baker, R. S., Gowda, S., Heffernan, N., & Heffernan, C. (2014). Population validity for Educational Data Mining models: A case study in affect detection. British Journal of Educational Technology, 45(3), 487–501. doi: 10.1111/bjet.12156
   Web of Science ®Google Scholar
• Ocumpaugh, J., Baker, R. S., S., Pedro, M. O., Hawn, M. A., Heffernan, C., Heffernan, N., & Slater, S. A. (2017). Guidance counselor reports of the ASSISTments college prediction model (ACPM). In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 479–488), Vancouver, British Columbia. ACM. DOI:10.1145/3027385.3027435
   Google Scholar
• Olver, M. E., Stockdale, K. C., & Wormith, J. S. (2009). Risk assessment with young offenders: A meta-analysis of three assessment measures. Criminal Justice and Behavior, 36(4), 329–353. doi: 10.1177/009385480933145
   Web of Science ®Google Scholar
• Pattison, E., Grodsky, E., & Muller, C. (2013). Is the sky falling? Grade inflation and the signaling power of grades. Educational Researcher, 42(5), 259–265. doi: 10.3102/0013189x13481382
   PubMed Web of Science ®Google Scholar
• Pelánek, R. (2014). A brief overview of metrics for evaluation of student models. In Workshop Approaching Twenty Years of Knowledge Tracing (BKT20y) (p. 6). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/JEDM087
   Google Scholar
• Pelánek, R. (2015). Metrics for evaluation of student models. JEDM-Journal of Educational Data Mining, 7(2), 1–19.
   Google Scholar
• Perna, L., Lundy-Wagner, V., Drezner, N. D., Gasman, M., Yoon, S., Bose, E., & Gary, S. (2009). The contribution of HBCUs to the preparation of African American women for STEM careers: A case study. Research in Higher Education, 50(1), 1–23. doi: 10.1007/s11162-008-9110-y
   Web of Science ®Google Scholar
• Phillips, M., Yamashiro, K., Farrukh, A., Lim, C., Hayes, K., Wagner, N., … Chen, H. (2015). Using research to improve college readiness: A research partnership between the Los Angeles Unified School District and the Los Angeles Education Research Institute. Journal of Education for Students Placed at Risk, 20(1–2), 141–168. doi: 10.1080/10824669.2014.990562
   PubMed Web of Science ®Google Scholar
• Piety, P. J., Hickey, D. T., & Bishop, M. (2014). Educational data sciences: Framing emergent practices for analytics of learning, organizations, and systems. Paper presented at the Proceedings of the Fourth International Conference on Learning Analytics and Knowledge. DOI:10.1145/2567574.2567582
   Google Scholar
• Quadri, M. M., & Kalyankar, N. V. (2010). Drop out feature of student data for academic performance using decision tree techniques. Global Journal of Computer Science and Technology, 10(2), 2–5. Retrieved from https://computerresearch.org/index.php/computer/article/view/891
   Google Scholar
• R Development Core Team. (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
   Google Scholar
• Renzulli, J. S., & Park, S. (2000). Gifted dropouts: The who and the why. Gifted Child Quarterly, 44(4), 261–271. doi: 10.1177/001698620004400407
   Web of Science ®Google Scholar
• Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen's d, and r. Law and Human Behavior, 29(5), 615–620. doi: 10.1007/s10979-005-6832-7
   PubMed Web of Science ®Google Scholar
• Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J. C., & Müller, M. (2011). pROC: an open-source package for R and S + to analyze and compare ROC curves. BMC Bioinformatics, 12(1), 77. doi: 10.1186/1471-2105-12-77
   PubMed Web of Science ®Google Scholar
• Robnett, R. D., & Leaper, C. (2013). Friendship groups, personal motivation, and gender in relation to high school students' STEM career interest. Journal of Research on Adolescence, 23(4), 652–664. doi: 10.1111/jora.12013
   Web of Science ®Google Scholar
• Rumberger, R. W., Addis, H., Allensworth, E., Balfanz, R., Duardo, D., Dynarksi, M., … Tuttle, C. (2017). Preventing dropout in secondary schools (NCEE 2017-4028). Washington, DC. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/wwc_dropout_092617.pdf
   Google Scholar
• Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. doi: 10.1177/0002764213498851
   Web of Science ®Google Scholar
• Sithole, A., Chiyaka, E. T., McCarthy, P., Mupinga, D. M., Bucklein, B. K., & Kibirige, J. (2017). Student attraction, persistence and retention in STEM Programs: Successes and continuing challenges. Higher Education Studies, 7(1), 46–59.
   Google Scholar
• Soland, J. (2013). Predicting high school graduation and college enrollment: Comparing early warning indicator data and teacher intuition. Journal of Education for Students Placed at Risk (JESPAR), 18(3–4), 233–262. doi: 10.1080/10824669.2013.833047
   Google Scholar
• Soland, J. (2017). Combining academic, noncognitive, and college knowledge measures to identify students not on track for college: A data-driven approach. Research & Practice in Assessment, 12 Retrieved from http://ezproxy.cul.columbia.edu/login?url= https://search.proquest.com/docview/1928643765?accountid=10226
   Google Scholar
• Stuit, D., O’Cummings, M., Norbury, H., Heppen, J., Dhillon, S., Lindsay, J., & Zhu, B. (2016). Identifying early warning indicators in three Ohio school Districts (REL 2016- 118). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest. Retrieved from http://ies.ed.gov/ncee/edlabs
   Google Scholar
• Sublett, C., & Gottfried, M. A. (2017). Individual and Institutional Factors of Applied STEM Coursetaking in High School. Teachers College Record, 119(10), n10.
   Web of Science ®Google Scholar
• Sullivan, W., Marr, J., & Hu, G. (2017). A predictive model for standardized test performance in Michigan schools. In Applied Computing and Information Technology (pp. 31–46). Cham: Springer.
   Google Scholar
• Supovitz, J. A., Foley, E., & Mishook, J. (2012). In search of leading indicators in education. Education Policy Analysis Archives, 20(19), 19–23.
   Google Scholar
• Swets, J. A. (1988). Measuring the accuracy of diagnostic systems. Science, 240(4857), 1285–1293. doi: 10.1126/science.3287615
   PubMed Web of Science ®Google Scholar
• Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological Science Can Improve Diagnostic Decisions. Psychological Science in the Public Interest, 1(1), 1–26. doi: 10.1111/1529-1006.001
   PubMedGoogle Scholar
• Swets, J. A., & Pickett, R. M. (1982). Evaluation of diagnostic systems: Methods from signal detection theory. New York, NY: Academic Press.
   Google Scholar
• Tamhane, A., Ikbal, S., Sengupta, B., Duggirala, M., & Appleton, J. (2014). Predicting student risks through longitudinal analysis. Paper presented at the Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, New York, NY. http://dl.acm.org/citation.cfm?id=2623355. DOI:10.1145/2623330.2623355.
   Google Scholar
• Torres, D. D., Bancroft, A., & Stroub, K. (2015). Evaluating high school dropout indicators and assessing their strength. Houston Education Research Consortium Research Brief, 3(5): 1–11.
   Google Scholar
• U.S. Department of Education. (2014; March 26). Academic competitiveness grants and national science and mathematics access to retain talent (smart) grants. Retrieved from https://www2.ed.gov/programs/smart/index.html
   Google Scholar
• Vivo, J. M., & Franco, M. (2008). How does one assess the accuracy of academic success predictors? ROC analysis applied to university entrance factors. International Journal of Mathematical Education in Science and Technology, 39(3), 325–340. doi: 10.1080/00207390701691566
   Google Scholar
• Whittaker, R. C. (2014). Teaching in context using a mobile phone scenario. Master thesis, University of Salford.
   Google Scholar
• Willis. (2013). Higher education in science, technology, engineering and mathematics: Science and Technology Committee Report — Motion to Take Note— in the House of Lords at 5:21 pm on 21st March 2013. Retrieved from https://www.theyworkforyou.com/lords/?id=2013-03-21a.759.1&s=Little+Aston, last accessed 6/24/2018.
   Google Scholar
• Wilson, J., Olinghouse, N. G., McCoach, D. B., Santangelo, T., & Andrada, G. N. (2016). Comparing the accuracy of different scoring methods for identifying sixth graders at risk of failing a state writing assessment. Assessing Writing, 27, 11–23. doi: 10.1016/j.asw.2015.06.003
   Web of Science ®Google Scholar
• Zwieg, M. H., & Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39(4), 561–577.
   PubMed Web of Science ®Google Scholar