Personalized Education through Individualized Pathways and Resources to Adaptive Control Theory-Inspired Scientific Education (iPRACTISE): Proof-of-Concept Studies for Designing and Evaluating Personalized Education

Figures & data

Fig. 1 Conceptualization of iPRACTISE as a control theory-inspired cruise control system that can be used by students, instructors, and program administrators to recommend optimal training pathways to minimize discrepancies between each student’s current and target ability levels.

Fig. 2 Summary of survey questions designed to help users specify their training goals and preferences; (B) a sample assessment question and progression of a hypothetical student’s estimated ability level during the assessment.

Fig. 3 Personalized post-assessment dashboard for a hypothetical student on Introduction to R, with: (A) a summary of the questions missed by the student and the correct responses; and (B) training materials ranked in descending order of fit to the student, with fit information across attributes (in terms of difficulty level, method of delivery, and estimated training time).

Fig. 4 (A) Scatterplot of the updated item easiness estimates compared to item easiness used initially in the BOAST adaptive test, which consisted of a combination of estimates from the calibration sample and instructor’s expert ratings; and (B) histogram plots showing the distributions of standard errors (SEs) in ability estimates in the adaptive as compared to the calibration groups. Students who were exposed to a similar range of number of items (between 8 and 10 items) across the two test groups were highlighted with a darker shade, and their respective averages in SE estimates are depicted in text.

Fig. 5 Screenshots of selected instructor functionalities in iPRACTISE, including: (A) a dashboard for viewing students’ progress; and (B) an assessment portal for browsing through and selecting existing assessment items from the test archive, as well as adding new items.

Table 1 Comparisons of fit across several alternative Item Response Theory models of individuals’ response patterns in the Introduction to R course.

Fit measures	Rasch unidimensional model (Model 1)	2 PL model, unidimensional model (Model 2)	2PL model unidimensional model with equality constraints (Model 3)	Bifactor model with one specific factor (Model 4)
AIC	2070.44	2067.57	2056.58	2062.16
BIC	2234.39	2390.18	2299.86	2350.40
SABIC	2038.53	2004.78	2009.23	2006.07
R ${}^2$	0.32	0.49	0.47	0.46, 0.06

NOTE: AIC $=$ Akaike Information Criterion (Akaike 1973); BIC $=$ Bayesian Information Criterion (Schwarz 1978); SABIC $=$ sample size-adjusted BIC (Sclove 1987; Tofghi and Enders 2007); 2PL model $=$ 2-parameter logistic model.

Table 2 Summary statistics for the rated importance (on a scale of 0–100) of attributes hypothesized to help participants select training materials.

Attributes	Mean	SD
AbilityFit	81.97	19.96
TopicFit	80.05	21.10
Method	71.67	19.54
TrainingTime	65.49	21.56
VisualAppeal	51.22	28.09

NOTE: AbilityFit $=$ Fit of the material’s difficulty level relative to the user’s ability level; TopicFit $=$ Fit of the topic given their interests; Method $=$ Fit of the method of delivery; TrainingTime $=$ Fit of the training time; and Visual appeal $=$ Visual appeal of the training material.

Table 3 Summary statistics for the rated usefulness (on a scale of 0–100) of selected iPRACTISE features.

Features	Mean	SD
TrainingResource	76.12	23.68
SummaryMissedQs	74.46	25.57
PersonalizedCourseTree	70.56	26.16
RankOrder	69.91	25.89
TimelyAssessment	65.72	25.37
GraphicalFit	59.79	27.88
CourseTree	59.51	28.16

NOTE: TrainingResource $=$ Availability of training resources; SummaryMissedQs $=$ Table of missed responses; PersonalizedCourseTree $=$ Personalized course tree based on assessment results; RankOrder $=$ rank ordering of the fit of the training materials for the user; TimelyAssessment $=$ Availability of timely assessments; GraphicalFit $=$ Radar plot of the fit of a training material relative to the participant across domains; CourseTree $=$ Course tree displaying topics, subtopics, and available resources.

Akaike, H. (1973), “Information Theory and an Extension of the Maximum Likelihood Principle,” in Second International Symposium in Information Theory, eds. B. N. Petrov, and F. Csáki, pp. 267–281, Budapest: Akadémiai Kiadó. DOI: 10.1007/978-1-4612-1694-0_15.

 Google Scholar

Schwarz, G. (1978), “Estimating the Dimension of a Model,” The Annals of Statistics, 6, 461–464. DOI: 10.1214/aos/1176344136.

 Web of Science ®Google Scholar

Sclove, L. S. (1987), “Application of Model-Selection Criteria to Some Problems in Multivariate Analysis,” Psychometrika, 52, 333–343. DOI: 10.1007/BF02294360.

 Web of Science ®Google Scholar

Tofghi, D., and Enders, C. K. (2007), “Identifying the Correct Number of Classes in Mixture Models,” in Advances in Latent Variable Mixture Models, eds. G. R. Hancock and K. M. Samulelsen, pp. 317–341, Greenwich, CT: Information Age.

 Google Scholar

Data Availability Statement

Data analysis scripts and de-identified data for which students have given consent to share their data to enhance the designs, developments, and delivery of statistics, mathematics, and data science education are openly available in the Open Science Framework at osf.io/vr43t. Other historical data collected in previous classes and used for calibration of assessment items within iPRACTISE are not shared when student consent was not obtained previously and the data could contain identifying information.