Discrimination with unidimensional and multidimensional item response theory models for educational data

Figures & data

Table 1. Data description.

Question	Area	Percentage correct
Q1	Basic statistics	93.3
Q2	Basic statistics	92.9
Q3	Basic statistics	54.6
Q4	Basic statistics (graphical)	79.8
Q5	Contingency table	96.6
Q6	Contingency table	60.5
Q7	Contingency table	75.2
Q8	Binomial distribution	79.0
Q9	Normal distribution	77.3
Q10	Normal distribution	60.1
Q11	Testing of hypothesis	52.5
Q12	Testing of hypothesis	42.4
Q13	Testing of hypothesis	87.0
Q14	Regression	75.6
Q15	Regression	87.0
Question	Area	Percentage in each category
		0	1	2	3	4	5
Q16	Chi-square test	9.7	2.5	3.4	5.0	27.7	51.7

Figure 1. The graphical representation of a unidimensional and a multidimensional model with within-item and with between-item multidimensionality. The left panel represents the univariate model. The middle panel depicts a multidimensional model with within-item multidimensionality, and the right panel indicates a multidimensional model with between-item multidimensionality.

Figure 2. Depicts the poly-choric correlations of 16 questions. Assuming that there are latent continuous variables underlying the categorical variables which are normally distributed, poly-choric correlation estimates the correlation between the assumed underlying continuous variables. Blue color squares indicate positive correlation: a darker color means a stronger correlation. Light-pink color squares indicate a weak negative correlation.

Table 2. Eigenvalue and explained variance for principal component analysis.

Component	Eigenvalue	Explained variance
		%	Cumulative%
1	6.54	40.90	40.90
2	1.42	8.87	49.77
3	1.32	8.24	58.01
4	1.21	7.59	65.60
5	1.07	6.72	72.32
6	0.89	5.60	77.92
…	…	…	…

Table 3. Global fit statistic for each model up to 3 dimensions.

M2PL + MGR
Dimensions	p	AIC	AICc	BIC	SABIC
D1	36	3823.35	3836.60	3948.35	3834.25
D2	51	3830.24	3858.76	4007.33	3845.68
D3	65	3824.04	3873.92	4049.73	3843.71
M3PL + MGR
Dimensions	p	AIC	AICc	BIC	SABIC
D1	51	3825.61	3854.12	4002.69	3841.03
D2	66	3824.60	3876.32	4053.77	3844.57
D3	80	3824.97	3907.51	4102.75	3849.17
M4PL + MGR
Dimensions	p	AIC	AICc	BIC	SABIC
D1	66	3833.98	3885.71	4063.16	3853.95
D2	81	3824.78	3909.93	4106.03	3849.28
D3	95	3823.37	3951.81	4153.23	3852.11

$p=$ number of parameters to estimate, AIC: Akaike information criteria; AICc: Sample size corrected AIC; BIC: Bayesian information criteria; SABIC: Sample size corrected BIC. D1, D2 and D3 represent models with 1, 2 and 3 dimensions, respectively.

Table 4. P-values for likelihood ratio tests.

	M3PL + MGR				M2PL + MGR
M2PL + MGR		D1	D2	D3		D1	D2	D3
	D1	0.0232	0.0013	0.0000	D1		0.0819	0.0013
	D2		0.0020	0.0002	D2	0.0819		0.0019
	D3			0.0157	D3	0.0013	0.0019
	M4PL + MGR				M3PL + MGR
M2PL + MGR		D1	D2	D3		D1	D2	D3
	D1	0.0144	0.0001	0.0000	D1		0.0088	0.0000
	D2		0.0002	0.0000	D2	0.0088		0.0159
	D3			0.0008	D3	0.0000	0.0159
	M4PL + MGR				M4PL + MGR
M3PL + MGR		D1	D2	D3		D1	D2	D3
	D1	0.1182	0.0007	0.0000	D1		0.0010	0.0000
	D2		0.0126	0.0008	D2	0.0010		0.0090
	D3			0.0073	D3	0.0000	0.0090

Table 5. Two- and three-factor solution for each model.

Question	Area	M2PL + MGR		M3PL + MGR		M4PL + MGR
		D2	D3	D2	D3	D2	D3
Q 1	Basic statistics	1	3	2	2	1	1
Q 2	Basic statistics	1	1	1	3	2	3
Q 3	Basic statistics	1	1	1	3	1	1
Q4	Basic statistics (graphical)	1	3	2	2	1	1
Q5	Contingency table	1	1	1	1	1	2
Q6	Contingency table	1	1	1	1	1	2
Q7	Contingency table	1	1	2	1	1	2
Q8	Binomial distribution	1	1	1	3	2	3
Q9	Normal distribution	1	1	1	1	2	3
Q10	Normal distribution	1	1	1	1	2	3
Q11	Testing of hypothesis	2	2	2	1	1	2
Q12	Testing of hypothesis	2	2	1	1	1	2
Q13	Testing of hypothesis	1	1	1	1	1	2
Q14	Regression	1	1	1	3	2	3
Q15	Regression	1	1	1	3	2	3
Q16	Chi-square test	1	1	1	3	2	3

The questions in black color represent the $1^{st}$ factor, red the $2^{nd}$ factor and similarly green color questions represent the $3^{rd}$ factor.

Figure 3. Item characteristic curves (ICCs) for factor 1. The black line represents the item curve of unidimensional 3PL + GR and red line represents the M4PL + MGR model.

Figure 4. Item characteristic curves (ICCs) for factor 2. The black line represents the item curve of unidimensional 3PL + GR and red line represents the M4PL + MGR model.

Figure 5. Item characteristic curves (ICCs) for dichotomous items in factor 3. The black line represents the item curve of unidimensional 3PL + GR and red line represents the M4PL + MGR model.

Figure 6. Item characteristic curves (ICCs) for graded response item Q₁₆. The black line represents the item curve of unidimensional 3PL + GR and red line represents the M4PL + MGR model.

Figure 7. Data generation (simulation model, the true model) and data analysis (fitted model).

Table 6. Simulation results for discrimination parameter (mean RMSE, AB and bias over 16 items and 1000 simulation runs) and AIC (mean over 1000 simulation runs).

		Analysis model	Discrimination parameter
True model	N		RMSE		AB		Bias		AIC
			Mean	SD	Mean	SD	Mean	SD	Mean
2PL+	300	2PL + GR	0.341	0.199	0.256	0.130	0.051	0.058	4773
GR		M2PL + MGR	0.363	0.192	0.271	0.127	0.075	0.089	4776
	500	2PL + GR	0.242	0.117	0.188	0.086	0.026	0.026	7942
		M2PL + MGR	0.260	0.115	0.203	0.086	0.047	0.073	7946
	1000	2PL + GR	0.165	0.075	0.130	0.056	0.013	0.012	15830
		M2PL + MGR	0.182	0.079	0.144	0.062	0.033	0.069	15836
M2PL+	300	2PL + GR	0.415	0.285	0.310	0.205	−0.079	0.258	4740
MGR		M2PL + MGR	0.410	0.278	0.295	0.171	0.073	0.103	4734
	500	2PL + GR	0.305	0.213	0.252	0.206	−0.061	0.254	7872
		M2PL + MGR	0.285	0.167	0.217	0.116	0.043	0.064	7860
	1000	2PL + GR	0.233	0.215	0.199	0.217	−0.075	0.252	15479
		M2PL + MGR	0.191	0.104	0.149	0.078	0.023	0.040	15468

Table 7. Simulation results for difficulty parameter (mean RMSE, AB and bias over 16 items and 1000 simulation runs).

True model		Analysis model	Difficulty parameter
	N		RMSE		AB		Bias
			Mean	SD	Mean	SD	Mean	SD
2PL+	300	2PL + GR	0.336	0.299	0.244	0.182	0.063	0.096
GR		M2PL + MGR	0.343	0.302	0.248	0.184	0.078	0.100
	500	2PL + GR	0.238	0.176	0.183	0.124	0.027	0.042
		M2PL + MGR	0.242	0.178	0.185	0.125	0.041	0.046
	1000	2PL + GR	0.161	0.110	0.125	0.081	0.016	0.023
		M2PL + MGR	0.164	0.112	0.127	0.082	0.029	0.027
M2PL+	300	2PL + GR	0.332	0.279	0.243	0.175	0.049	0.087
MGR		M2PL + MGR	0.341	0.275	0.250	0.174	0.067	0.088
	500	2PL + GR	0.239	0.179	0.182	0.122	0.021	0.050
		M2PL + MGR	0.245	0.181	0.186	0.122	0.038	0.049
	1000	2PL + GR	0.161	0.103	0.127	0.079	0.017	0.022
		M2PL + MGR	0.165	0.104	0.129	0.079	0.024	0.023

Data availability

The data that support the findings of this study are available from the corresponding author (Ul Hassan) upon reasonable request.