Assessing Instructional Modalities: Individualized Treatment Effects for Personalized Learning

Figures & data

Table 1. Study inputs from the SDSU student information database.

	Descriptives for	Descriptives for
Inputs	face-to-face offering	online offering
Online/Traditional indicator	157 students	57 students
Semester enrolled	Fall 07, 08, 09, 12	Summer 2013
Last Stat course taken	See Figure 1	See Figure 1
Grade in last Stat course	3.2 (0.8)	3.0 (0.9)
SDSU GPA	3.0 (0.6)	3.0 (0.6)
Total GPA	3.1 (0.5)	3.1 (0.5)
Student Level	See Figure 1	See Figure 1
Major	See Figure 1	See Figure 1
Pre-major indicator
Admission basis (first-time-freshman)	55%	47%
Units attempted that semester	12.5 (3.3)	7.1 (4.1)
Units earned that semester	12.1 (3.3)	6.9 (3.7)
Total units attempted	110.5 (38.4)	114.5 (36.5)
Total units earned	110 (31)	109 (33)
Total # online course units previously	2.3 (median 0)	3.6 (median 3)
Age	24.4 (6.0)	25.1 (6.8)
Gender (female)	30%	30%
Ethnicity	22% URM	28% URM
Low income	37%	0%
Equal opportunity program (EOP)	18%	12%
1st generation college student	13%	14%
County resided
Previous institution

NOTE: SDSU places students in a pre-major prior to completing lower division general elective requirements as well as pre-requisites for a given major (pre-major indicator). Admission basis categorizes students as entering SDSU as first-time freshmen (FTF) or transfer students and as California residents, out-of-state, or international students. The CSU equal opportunity program (EOP) is designed to improve access and retention to historically low-income and educationally disadvantaged students. For continuous inputs, the average value is reported with standard deviation in parentheses unless otherwise stated.

Figure 1. The bar charts present relative frequencies of class level, majors, and previous statistics course taken. The side-by-side boxplot presents final exam scores as a percentage. All graphs are stratified by enrollment in the online or traditional course. The major categories on the second bar chart of the top row are math, computer science (CS), psychology (Psy), statistics (Stat), biology/physics (B/P), arts & letters (A&L), business (CBA), and other. On the bar chart in the bottom row, JC denotes junior or community college.

Table 2. Variable importance ranking (in rank order) for random forest fit of final exam score on inputs for students in the online class (left) and for students in the traditional sections (right).

Online offering		Traditional offering
	Direction of		Direction of
Input	association	Input	association
GPA	+	GPA	+
Grade in last Stat course	+	Major
EOP	–	Total # units attempted	–
Admission basis		Total # units earned	+
Major		Grade in last Stat course	+
Total # units attempted	–	Age	–

NOTE: The direction of association identifies either a positive or negative trend for each continuous and binary variable.

Table 3. Non-zero coefficients from lasso fit of final exam score on inputs for students in the online class (left) and for students in the traditional sections (right).

Online offering		Traditional offering
Input	Lasso coefficient	Input	Lasso coefficient
GPA	0.10 (0.02)	GPA	0.07 (0.04)
First generation	$-$0.04 (0.05)	Term units attempted	0.04 (0.02)
Online units	0.04 (0.02)	Online units	0.05 (0.02)
Major (CS)	$-$0.01 (0.04)
Age	0.02 (0.01)
First-time freshman	$-$0.002 (0.03)

NOTE: Standard errors in parentheses are obtained from the selectiveInference R package (Tibshirani, et al. 2016).

Table 4. Average individualized treatment effects for final exam score (percentage points) for students enrolled in the online class and in the standard face-to-face lecture classes.

	Online offering	Standard offering
Random forest	14	1
Lasso	7	2

Table 5. Inputs distinguishing students who benefited from the online offering (ITE Top 20%) in terms of final exam score.

		ITE Top 20%	Comp gp
	p-value	n=43	n=43
Final Exam
Term Units Attempted	0.003	9.4	12.2
Term Units Earned	0.005	9.0	11.7
Age	0.11	25.6	23.2
Online Units	0.06	2.2	3.8
First Generation	0.12	30%	49%
Admission Basis	0.04
First Time Freshman		44%	70%
Upper Division Transfer		44%	38%
Graduate		12%	2%

NOTE: The p-values are determined from the appropriate t-test comparing the two groups on the given input. The latter two columns present average summaries on each input for the top 20% and comparison groups, respectively.

Figure 2. Percentage of majors (top) and grades (bottom) for students from the Stat 350A online class (11 students) and traditional sections (77 students) completing the follow-up course Stat 350B. The bottom graphic displays course grade in Stat 350B.

Tibshirani, R. J., Taylor, J., Lockhart, R., and Tibshirani, R. (2016), “Exact Post-Selection Inference for Sequential Regression Procedures,” Journal of the American Statistical Association, 111, 600–620.

 Web of Science ®Google Scholar