Item Response Theory and Confirmatory Factor Analysis: Complementary Approaches for Scale Development

Figures & data

Table 1. What can be learned about measures using CFA for ordinal level data

Model level analysis

1. Assessment of fit of the factor model based on an overall comparison of the analysis matrix to a new matrix implied by the model

2. Assessment of model fit based on a matrix of residuals – differences in values of corresponding elements of the analysis matrix and implied matrix.

Item level analysis

3. Estimate of the relationship between each indicator and the latent factor it measures in unstandardized and standardized metrics (factor loadings)

5. Estimates of cut points (thresholds) on the underlying theoretical normal distribution of the social isolation construct corresponding to the choice of different response options

6. Estimate of the amount of variance of each indicator that is explained and unexplained by an underlying factor

7. Estimates of relationships among error terms of observed indicators

8. Assessment of whether there are significant differences in parameter estimates across items, groups, time, or data collection methods

Scale level analysis

9. Confirmation of factor structure, including the number of factors, pattern of item loadings, and correlations among factors

10. Comparison of quality of competing alternative factor structures, such as second-order and bifactor structures

11. Estimates of the variance and mean of each factor in the model and weighted factor scores for individuals

12. Assessment of whether there are differences in means and variances of factors across subgroups of respondents

Table 2. What can be learned about measures using IRT for ordinal level data

Model level analysis

1. Assessment of model fit using indexes developed specifically for ordinal response IRT models

Item level analysis

2. Assessment of item fit

3. Estimates of IRT item slope parameters (a-parameters) and location (b-parameters) using marginal maximum likelihood methods

4. Assessment of item category response coverage: distribution of category response functions over θ

5. Assessment of item information: distribution of item information over θ

6. Assessment of differential item functioning across subgroups of respondent

Scale level analysis

7. Assessment of scale information, conditional standard errors, and conditional reliability

8. Estimates of person parameters (θ estimates, factor scores); assessment of score estimate precision using conditional standard errors

9. Estimates of expected true scores in original item metrics from model-based θ estimates

10. Assessment of differential scale functioning across subgroups of respondents

Table 3. Item response frequencies

Item	No, Never	Yes, Sometimes	Yes, Often
1. Do you ever wish you could run away from home?	433	156	37
	69.2%	24.9%	5.9%
2. Do you ever feel nobody cares about you?	330	229	67
	52.7%	36.6%	10.7%
3. Do you ever feel like you don’t know what to do?	165	373	88
	26.4%	59.6%	14.1%
4. Do you ever feel all alone?	278	253	95
	44.4%	40.4%	15.2%
5. Do you ever feel no one listens to you?	314	246	66
	50.2%	39.3%	10.5%

Figure 1. First-order factor model of the ESSP Social Isolation Scale

Table 4. CFA parameter estimates

Item	^Factor loadings (SE)	95% CIs	^^Factor loadings (S)	Item R^2	Thresholds
1. Do you ever wish you could run away from home?	1.00 (0.00)	–	0.65	0.42	.50 1.56
2. Do you ever feel nobody cares about you?	1.20 (.100)	[1.007, 1.400]	0.78	0.61	.07 1.24
3. Do you ever feel like you don’t know what to do?	0.92 (.094)	[0.735, 1.104]	0.60	0.36	−.63 1.08
4. Do you ever feel all alone?	1.11 (.080)	[0.958,1.270]	0.73	0.53	−.14 1.03
5. Do you ever feel no one listens to you?	1.18 (.101)	[0.982,1.378]	0.77	0.59	.00 1.25

^ = unstandardized; ^^ = standardized. SE = standard error. CI = confidence interval.

All loadings were statistically significant at the p < .001 level.

Table 5. Item generalized S-χ² and RMSEA indexes

Item	S-χ^2 (df)	p	RMSEA	95% CI
1. Do you ever wish you could run away from home?	30.77(10)	.00	.06	[.03, .08]
2. Do you ever feel nobody cares about you?	16.35(8)	.04	.04	[.00, .07]
3. Do you ever feel like you don’t know what to do?	11.12(9)	.27	.02	[.00, .05]
4. Do you ever feel all alone?	11.94(9)	.22	.02	[.00, .05]
5. Do you ever feel no one listens to you?	6.97(8)	.54	.00	[.00, .05]

Table 6. Item graded response model parameter estimates

Item	a (se)	b1 (se)	b2 (se)
1. Do you ever wish you could run away from home?	1.59 (.18)	.78 (.09)	2.29 (.21)
2. Do you ever feel nobody cares about you?	2.42 (.24)	.12 (.06)	1.57 (.12)
3. Do you ever feel like you don’t know what to do?	1.39 (.14)	−1.00 (.11)	1.71 (.15)
4. Do you ever feel all alone?	1.93 (.19)	−.17 (.07)	1.38 (.11)
5. Do you ever feel no one listens to you?	2.52 (.24)	.03 (.06)	1.57 (.11)

Figure 2. Category response curves for 3 response options (Ps) per item

P1 = probability of responding 1 (No, never); P2 = probability of responding 2 (Yes, sometimes); P3 = probability of responding 3 (Yes, always)

Figure 3. Item information curves

Information (θ) = conditional scale information function for each item. Information values are not bounded by a value of 1, they can be >1 at points along θ.

Figure 4. Scale information curve and conditional standard error curve

Information (θ) = conditional-scale information function (solid line); Standard Error (θ) = conditional scale standard errors (dotted line). The curves are mathematical functions of each other where Standard Error (θ) = 1 – √ Information (θ). Higher information along the θ scale gives rise to lower standard errors resulting in more precise θ estimates.

Figure 5. Conditional reliability

Reliability (θ) = conditional-scale reliability function. Higher values along the θ scale indicate more reliable score estimates. Information, conditional standard errors, and conditional reliability are mathematically related.

Figure 6. Scale characteristic curve linking estimated θ scores and expected true scores

Scale characteristic curves map model-based estimated θ scores to expected true scores. For example, an estimated θ score of 0 would translate into an expected true score of 8; an estimated θ score of 1 would translate into an expected true score of 10. As noted, these θ transformation scores provide model-based estimated true scores that are in the original scale score metric.

Table 7. Integrated performance information about the social isolation scale based on CFA and IRT results

CFA	IRT
Model level analysis	Model Level Analysis
1. The Social Isolation scale had overall good fit according to multiple indices.	The graded response model for the scale had good fit according to multiple indices.
2. The scale had good fit across all elements of the analysis matrix based on the residual matrix.
Item level analysis	Item level analysis
	3. Individual items had good fit based on S-χ^2 p-values and S-χ^2-RMSEA values.
4. Unstandardized factor loadings indicated that scores on items were responsive to 1-unit changes in the social isolation construct; item scores changed between .92 and 1.20 units on a 3-point scale.	4. Slope parameters indicated items were able to differentiate at high and very high levels among students with different levels of social isolation. Location parameters indicated that all items except item 3 measured higher levels of social isolation (θ) better than lower levels
5. According to item thresholds, only two of the five items (3 & 4) captured below-average levels of social isolation (between −3.0 and 0). For the three other items, latent social isolation values from −3.0 to 0 or above 0 were all captured in the first response option.	5. Response category probabilities tended to concentrate in higher levels of θ; this pattern suggested that it took higher levels of social isolation to endorse the “Yes, sometimes” or the “Yes, often” response categories
6. Because the social isolation construct explained less than 50% of the variance of items 1 and 3, they were not strong measures of the construct and contributed less to its definition.	7. All items contributed substantive information to the precision and reliability with which scores across values of social isolation could be estimated.
8. No correlations among item error terms (unexplained variances) were indicated, which is desirable.
9. We did not conduct invariance tests in this study.	9. We did not test for DIF in this study.
Scale level analysis
10. Results indicated the Social Isolation scale is unidimensional. Because the 5 social isolation items loaded on one factor, no alternative factor structures were tested.	11. High scale information values, low conditional standard errors, and high conditional reliability indicated that the scale operated best over the 0 ≤ θ ≤ +2 range of the social isolation scale (θ)
12. The social isolation factor had a statistically significant variance of .46 on a Z score scale, which means the factor captured meaningful differences in scores. The mean, standard deviation, and range of the factor scores were .015, .55, and −.791 to 1.5, respectively.	12. According to expected a posteriori (EAP) estimation of student θ scores (person parameters), the average estimated θ score was .00, the standard deviation was .86, and the range of scores was −1.31 to 2.37.
	13. We transformed estimated θ scores into model-based expected true scores so they could be understood in terms of the items’ observed metric. The average estimated true score was 8.68, the standard deviation was 3.27, and the range of scores was 5 to 15.
14. We did not conduct multiple group comparisons of the mean or variance of social isolation factor scores for this study.	14. We did not test for DTF for this study.