Understanding the alternative Mantel-Haenszel statistic: Factors affecting its robustness to detect non-uniform DIF

Abstract

Test-item bias has become an increasingly challenging issue in statistics and education. A popular method, the Mantel-Haenszel test, is used for detecting non-uniform differential item functioning (DIF) but requires constructing several performance tiers to maintain robustness. The alternative Mantel-Haenszel (AMH) test was developed within the last three decades as a proxy procedure requiring only two scoring tiers. However, there needs to be more information on how essential factors like comparison group sizes and question discrimination affect its ability to detect bias. This study investigates how item difficulty and discrimination and the examined ratio between the focal and reference groups impact the likelihood of the AMH test detecting DIF. A comprehensive simulation study was conducted where the test scores were generated under three conditions. There are three commonly used difficulty levels (easy, medium, and hard), two discrimination levels (referred to as 'low’ and 'high’), and three group comparison ratios (1:1, 2:1, and 5:1). The simulation study showed the AMH test’s detection rates comparable to those of other standardized procedures like the Breslow-Day (BD) and even better. The current study aims to investigate and determine the factors that affect AMH test detection behavior. The study concludes with an application involving collegiate-level test data comparing students across genders and majors.

Keywords:

• Differential item functioning
• non-uniform DIF
• discrimination
• item difficulty
• Breslow-Day
• Mantel-Haenszel

Introduction

Differential item functioning (DIF) is a widely employed assessment tool in fields such as quantitative psychology, educational measurement, business management, insurance, and healthcare (Holland and Wainer 1993). In large-scale assessments, the accurate detection of DIF is paramount to ensure that tests are fair and unbiased. Items that produce biased results can negatively impact examinees’ opportunities in education, employment, and other areas of life, particularly for minority or marginalized groups (Jensen 1980; Scheuneman and Bleistein 1989).

However, challenges arise when tests manifest undetected DIF are misdiagnosed due to detection procedure limitations. Such oversights can lead to biased assessments of individuals, resulting in incorrect conclusions about their abilities. The Mantel-Haenszel (MH) method, a popular tool for this purpose, has its own set of challenges. Given its widespread use in large-scale testing, it’s imperative to be aware of its limitations. For instance, the MH method may not be as adept at identifying non-uniform DIF compared to other techniques. This can lead to method discrepancies, especially when non-uniform DIF is present but goes undetected by the MH method (Hambleton and Rogers 1989). Furthermore, the efficacy of the MH test can be influenced by various factors, including purification procedures, data fit to the Rasch model, test contamination, and sample sizes (Guilera et al. 2013).

The implications of a Type I error, or a false positive, in DIF detection using the MH method, can be profound. It could result in incorrect conclusions about student ability differences in educational contexts, influencing curriculum modifications, resource allocation, or pedagogical strategies. Notably, when the number of score groups in the Mantel-Haenszel procedure is reduced, it can significantly elevate the Type I error rate, especially if the ability distributions of the reference and focal groups are disparate (Clauser, Mazor, and Hambleton 1994). In professional environments, such errors might lead to unwarranted changes in certification exams or introduce biases in recruitment decisions.

Understanding the distinction between non-uniform and uniform DIF necessitates clearly explaining the concepts. Non-uniform DIF occurs when the relationship between item performance and group membership varies across ability levels, meaning that the probability of a correct response between the reference and focal groups changes along the ability continuum. This Type of DIF is represented by non-parallel item characteristic curves (ICCs) (Walker 2011). On the other hand, uniform DIF occurs when the relationship between item performance and group membership remains constant across ability levels, indicating that the difference in the probability of a correct response between the reference and focal groups stays the same along the ability continuum. Uniform DIF is represented by parallel ICCs (Amaechi and Onah 2020; Walker 2011; Ayilara et al. 2022).

Several studies have shown that various conditions, such as significant differences in sample sizes, item performance, and ability distributions between groups of examinees, can tremendously affect how often the Mantel-Haenszel (MH) procedure detects non-uniform DIF correctly (Rustam, Naga, and Supriyati 2019; Herrera and Gómez 2008; Marañón, Garcia, and Costas 1997; Mazor, Clauser, and Hambleton 1992, 1994; Narayanan and Swaminathan 1994; Narayanon and Swaminathan 1996; Swaminathan and Rogers 1990).

Mazor, Clauser, and Hambleton (1994) attempted to determine whether a simple adjustment of the MH procedure could improve detection rates for items showing non-uniform DIF. The researchers developed an ingenious alternative Mantel-Haenszel (AMH) procedure. They suggested that by partitioning examinees based on the average test score, particularly separating the examinees into either a 'high’ or a 'low’ scoring group, the MH procedure could be repeated three times for each item, once for the total group, once for the high-scoring group, and again for the low-scoring group. Thus, it should detect both uniform and non-uniform DIF. The study results concluded that the procedure helps to increase detection rates without increasing the Type I error rate.

MH and BD procedures

The MH technique is a Chi-squared contingency table-based method that detects the differences between reference and focal groups on all test items one by one. The total test scores define the ability continuum, divided into k intervals that serve as a basis for matching the two groups’ members. A comparison of both groups at each interval of k is made through a 2 × 2 contingency table. The table’s rows represent the group membership (reference or focal), whereas the columns represent the correct or incorrect responses to the studied item, as shown in Table 1. The reference group refers to the group that an item is suspected of favoring. In contrast, the focal group refers to the group where the item is suspected to function differently (Holland and Thayer 1986; Rustam, Naga, and Supriyati 2019; Khoeruroh and Retnawati 2020; Tang, He, and Tu 2023).

Table 1. Summary of data from the j^th 2 × 2 contingency Table.

		Response to the studied item
		Correct	Incorrect	Total
Membership group	Reference (R)	$A_j$	$B_j$	$n_{\mathit{\text{Rj}}}$
	Focal (F)	$C_j$	$D_j$	$n_{\mathit{\text{Fj}}}$
	Total	$m_{1j}$	$m_{0j}$	$N_j$

In Table 1, $A_j,$ $B_j,$ $C_j,$ and $D_j$ are the number of examinees in each cell in the j^th stratum, $N_j$ represents the total number of examinees in the j^th stratum. $n_{\mathit{\text{Rj}}}$ and $n_{\mathit{\text{Fj}}}$ represents the numbers of examines in the reference and focal groups in the j^th stratum, respectively. $m_{1j}$ and $m_{0j}$ represents the number of examinees who answer the studied item correctly and incorrectly, respectively. According to Mantel and Haenszel (1959), the MH Chi-square test statistic is defined as: (1) $${\chi }_{\mathit{\text{MH}}}^2=\frac{{\left(\left|\sum _j{A}_j-\sum _{j}E(A_j)\right|-0.5\right)}^2}{\sum _j{\mathit{\text{Var}}(A}_j)}$$(1) where ${E(A}_j)=\frac{n_{\mathit{\text{Rj}}}\mathrm{ }{m}_{1j}}{N_j},$ and ${\mathit{\text{Var}}(A}_j)=\frac{m_{1j}\mathrm{ }{m}_{0j}\mathrm{ }{n}_{\mathit{\text{Rj}}}\mathrm{ }{n}_{\mathit{\text{Fj}}}}{N_j^2\left(N_j-1\right)}.$

The Mantel-Haenszel (MH) and Breslow-Day (BD) procedures are standard methods for detecting differential item functioning (DIF). Both methods have their respective strengths and weaknesses, which must be carefully considered when detecting non-uniform DIF. The MH procedure is widely utilized for its simplicity, practicality, and effectiveness in detecting uniform DIF. However, it may yield misleading results when applied to non-uniform DIF, particularly under conditions with unequal sample sizes, non-normal ability distributions, or complex models (Holland and Thayer 1986). Despite these limitations, the MH method remains popular because it is easy to implement, does not require iterative calculations, and often provides satisfactory results for uniform DIF detection. Researchers should be aware of these limitations when using the MH procedure and consider its appropriateness based on the characteristics of the data at hand (Li 2015).

The BD procedure is specifically designed for analyzing non-uniform DIF, as it assesses odds ratio heterogeneity trends. While the method has significantly contributed to detecting DIF, it has some weaknesses, such as reduced accuracy compared to alternative methods like logistic regression (Aguerri et al. 2009). Additionally, the BD procedure can be sensitive to test assumptions, such as the assumption of equal variance between groups, which may only sometimes hold in practice (Penfield 2003). However, it is often combined with other methods, like the MH procedure, to improve overall performance and accuracy in detecting DIF.

The BD Chi-square statistic has an approximate Chi-square distribution with k-1 degrees of freedom under the null hypothesis. For a 2 × 2 × k contingency table, the BD Chi-square statistic is as follows: (2) $${\chi }_{\mathit{\text{BD}}}^2=\sum _j\frac{{\left(A_j-E\left(A_j|{\mathit{\text{OR}}}_{\mathit{\text{MH}}}\right)\right)}^2}{\mathit{\text{Var}}\left(A_j|{\mathit{\text{OR}}}_{\mathit{\text{MH}}}\right)}$$(2) where ${\mathit{\text{OR}}}_{\mathit{\text{MH}}}$ is the MH estimator of common odds ratio. $E\left(A_j|{\mathit{\text{OR}}}_{\mathit{\text{MH}}}\right)$ and $\mathit{\text{Var}}\left(A_j|{\mathit{\text{OR}}}_{\mathit{\text{MH}}}\right)$ are the mean and variance of $A_j$ under the null hypothesis, respectively. For more details, please see Breslow and Day (1980).

Aguerri et al. (2009) used the BD procedure to detect non-uniform DIF when one group’s average ability was considerably larger than another group. They used the BD procedure to effectively determine factors that affect the non-uniform DIF detection rate. The BD procedure results were compared with logistic regression analysis (LR) and the standard MH procedure. The researchers found that the BD was better than the LR and the MH procedure. The method was also used to test several other parameters, including the sample size and item, to effectively determine factors that affect the BD procedure’s use to detect non-uniform DIF. According to Aguerri et al. (2009), when the item with the greater discrimination and difficulty parameters for equally sized groups was omitted from the goodness-of-fit to the binomial distribution, the test returned a Type I error similar to the nominal one.

Penfield (2003) performed a DIF to compare a single reference group and multiple focal groups in another study. According to Penfield, conducting a separate DIF test had several undesirable qualities, which included an increase in Type I error rate and the need for substantial time and computing resources. He suggested using a procedure with the ability to assess DIF across multiple groups to address these drawbacks. In this regard, the researcher conducted a study to compare three MH procedures: the MH Chi-square statistic with no adjustment to the alpha level, the MH Chi-square statistic with a Bonferroni adjusted alpha level and the generalized Mantel-Haenszel (GMH) statistic. From the study’s findings, GMH was the most appropriate method for assessing DIF among multiple groups.

Building on his earlier exploration of DIF methodologies, Penfield (2003) conducted a simulation study to evaluate the statistical power and the rate of Type I error of BD and combined decision rule (CDR). The CDR method combines BD with the MH Chi-square test. The results showed that the CDR performance was better under various conditions regarding the Type I error rate and the statistical power compared to other methods. The results also indicate that CDR might be more accurate regarding DIF detection compared to other DIF detection methods that are commonly used.

AMH procedure

Since the development of the AMH procedure, studies investigating it have been rare; Mellenbergh and Fidalgo Aliste (1997) compared the AMH procedure’s performance to that of the standard MH and iterative logit procedures. They found that the AMH procedure had a higher power rate than the other two procedures but later cautioned that the AMH procedure needed to be more robust. Their study showed significant relationships between sample size and DIF's effect on the performance of the AMH procedure. Several limitations surrounding the study nevertheless warrant further investigation. For instance, while the study considered two sample sizes (200 and 1,000), it conditioned the compared groups to have equal sizes. Additionally, the study investigated non-uniform DIF as a composite measure of both difficulty and discrimination. Finally, the researchers investigated symmetric non-uniform DIF, thus leaving gaps concerning how these factors affect asymmetric non-uniform DIF.

Building on these foundational studies, the present research is an extension of the study done by Mazor, Clauser, and Hambleton (1994). It uses a predictive model to explain the rate at which a non-uniform DIF item is detected when partitioning subjects by high and low ability levels based on the average test score. In this respect, the study aims to achieve several objectives to understand DIF and approaches used to detect DIF. There are three research questions in this study. They are:

1. How sample size ratio, item difficulty, and item discrimination affect non-uniform DIF detection using the AMH procedure?

2. What factors affect the rate of detection of non-uniform DIF using the AMH procedure?

3. What are the conditions under which the AMH procedure works best in detecting non-uniform DIF compared to the BD procedure?

There were several predictors used in these models. They include item difficulty, item discrimination, and sample size ratios. As an extension to Mazor, Clauser, and Hambleton (1994), the results also considered the case where the ability level distributions for the reference and focal groups were equal and unequal. Thus, this is also research to compare to the BD procedure described in Penfield (2003). Ultimately, this study identified the significant factors contributing to the detection rates and the conditions in which the highest non-uniform detection rates occurred. This study concludes by applying the AMH and BD procedures on test data collected at a university-level introductory statistics course.

Method

The study combines methods described by Mazor, Clauser, and Hambleton (1994) and Penfield (2003), which used AMH and BD procedures. Those procedures are contingency table and non-model-based procedures, suitable for detecting non-uniform DIF discussed in the current study. There are three processes: simulating the data, assessing the non-uniform DIF items, and creating the regression models, which were conducted in SAS software version 9.4, using several procedures such as PROC IRT and PROC LOGISTIC. Each step was explained in the following subsections.

Simulating the data

Each simulation contains the item scores for a 75-item examination, in which the last three items are created to contain non-uniform DIF representing the studied items. The number of items is considered since it reflects similar large-scale assessments. For non-studied items (72 items), difficulty levels were generated from a normal distribution with a mean of zero and a standard deviation of one. The discrimination levels were generated from a lognormal distribution with a mean of 0 and a standard deviation of .35. These considerations are based on a similar study by Penfield (2003). The guessing parameter for the non-studied items is .2. This set was also used for performance comparisons.

Parameters for the studied items are manipulated using the item response theory (IRT) procedure in SAS software to create non-uniform DIF based on six combinations between item discrimination and difficulty. It is worth mentioning that the three-parameter IRT model equation includes a scaling constant (D), which is usually set to D = 1.7 or D = 1 based on if the normal ogive or logistic metric is desired. In this study, the scaling constant (D) equals one. For more details regarding the scaling constant, interested readers are referred to Lord (2012).

Two levels for item discrimination were tested, including low ($a_r=.46$ and $a_f=.80$) and high ($a_r=.70$ and $a_f=1.97$). Three levels for item difficulty were considered; easy ($b=-1.50$), medium ($b=0$), and difficult ($b=1.50$). These parameters are taken from Mazor, Clauser, and Hambleton (1994). The guessing parameter, like the non-studied items, is set at .2. Three sample size ratios are considered in this study, namely 1:1 ($n_r=n_f=1000$), 2:1 ($n_r=1000,n_f=500$), and 5:1 ($n_r=1000,n_f=200$). These ratios are similar to those from Penfield (2003).

A total of 10,000 simulations were performed for each of the eighteen combinations of item discrimination, item difficulty, and reference-to-focal group ratio. These simulations are then stratified across two ability-level distribution conditions. The first case includes reference and focal groups from normal distributions with a mean of zero and a standard deviation of one. In the second case, the reference group’s ability levels remain unchanged, with the focus group having a mean of one standard deviation lower than the reference group. Based on the observed mean score, the simulation samples were partitioned into low and high-ability groups. The methods used in the studies by Penfield (2003) and Mazor, Clauser, and Hambleton (1994) are similar to the present method.

Assessing the non-uniform DIF

After creating the simulated item scores, the AMH and BD procedures were performed to determine whether non-uniform DIF is presented in the studied items. A significant level of 5% was considered in the current study for the following reasons: most quantitative research studies use a 5% level of significance. Secondly, it was proven that the Type I error rates were consistent at or below the nominal level of 5%, whether the group ability distributions were equal or unequal (Penfield 2003).

Type I error rates

In DIF studies, the Type I error (false positive) rate is defined as the percentage of simulations in which a null item is detected as having DIF while the item is free of DIF. Similar to detection (power) rates, Type I error rates for non-studied items were evaluated as a function of sample size ratios, item discrimination levels, item difficulty levels, and the reference and focal group ability distribution equality.

Creating the regression Models

Four logistic regression models were created to assess the likelihood of detecting non-uniform DIF from one of the studied items. Each of the two models will assess the DIF using the AMH and BD procedures, where each distinguishes between equal and unequal ability distributions. Each logistic regression model uses item discrimination, item difficulty, and group ratio as explanatory variables to predict the likelihood of detecting non-uniform DIF. The logit form of the logistic regression model is in the following form: (3) $$\mathit{\text{logit}}\left(p\right)={\beta }_0+{\beta }_1\left(\mathit{\text{groupratio}}\right)+{\beta }_2\left(\mathit{\text{discrimination}}\right)+{\beta }_3\left(\mathit{\text{dif}}\mathrm{fi}\mathit{\text{culty}}\right)+{\beta }_4\left(\mathit{\text{groupratio}}\times \mathit{\text{discrimination}}\right)+{\beta }_5\left(\mathit{\text{groupratio}}\times \mathit{\text{dif}}\mathrm{fi}\mathit{\text{culty}}\right)+{\beta }_6\left(\mathit{\text{discrimination}}\times \mathit{\text{dif}}\mathrm{fi}\mathit{\text{culty}}\right)+{\beta }_7\left(\mathit{\text{groupratio}}\times \mathit{\text{discrimination}}\times \mathit{\text{dif}}\mathrm{fi}\mathrm{c}\mathit{\text{ulty}}\right)$$(3)

$\mathit{\text{logit}}\left(p\right)=\mathrm{\text{log}}\left(\frac{p}{1-p}\right)$ represents the log of the odds of detecting non-uniform DIF. Model regression estimates are produced from the logistic model. Akaike information criterion (AIC), Bayesian information criterion (BIC), C-statistic, and correct classification rate (CCR) were reported to provide the model’s diagnostic statistic.

In the current study, both information criteria, AIC and BIC, are used for model selection. These statistics give two different methods of adjusting the −2 log-likelihood statistic for the number of model parameters and the sample size used. According to Portet (2020), the AIC is defined as: (4) $$\mathit{\text{AIC}}=–2\mathrm{ }\mathrm{\text{log}}\left(L\right)+2\left(p\right)$$(4) where $L$ is the maximized value of the likelihood function of the model, and $p$ is the number of the model’s parameters. Likewise, the BIC can be defined as: (5) $$\mathit{\text{BIC}}=–2\mathrm{ }\mathrm{\text{log}}\left(L\right)+p\left(\mathrm{\text{log}}\hspace{0.17em}(n)\right)$$(5)

where $n$ is the number of observations in the model (Wit, Heuvel, and Romeijn 2012). For further information regarding other measures of the model fit, such as C-statistic and CCR, refer to Caetano, Sonpavde, and Pond (2018) and Press and Wilson (1978), respectively.

Participants and application study

This study used data obtained from the Department of Student Experience and the Department of Mathematics, Statistics, and Physics at a national university in Qatar. The sample consisted of 1040 undergraduate students enrolled in Introduction to Statistics course. Students took an exam containing 35 multiple-choice questions, of which five questions were selected for DIF analysis for illustration purposes.

Demographic information, such as test scores, gender, and major, was collected to classify the students into different membership groups. According to Table 2, based on gender, 49.42% of the students with a test score average of 71.16 (SD = 12.52) were males, while 50.58% with a test score average of 72.65 (SD = 12.59) were females. In terms of their majors, students were classified into two main categories: STEM and non-STEM majors. STEM major students represented 33.46% of the sample, while non-STEM students represented 66.54%. The average test score of STEM and non-STEM major students was 73.75 (SD = 14.63) and 70.98 (SD = 11.47), respectively.

Table 2. Demographics of application data.

Groups	Percentages	Average Test Score (Out of 100)	SD
By Gender
Male	49.42%	71.16	12.52
Female	50.58%	72.65	12.59
By Majors
STEM	33.46%	73.75	14.63
non-STEM	66.54%	70.98	11.47

Analysis and results

Simulation study

The Type I errors of the AMH and the BD procedures, which refers to the percentage of simulations in which the AMH and the BD procedures successfully detected the no DIF (null) items, are shown in Tables 3 and 4. Both tables reflect percentages as a function of RF, AL, and BL, but the two tables contrast by considering equal and unequal ability distributions.

Table 3. Detection rates from equal ability distributions for no DIF items (type I error rates).

R:F Ratio	Discrimination	Difficulty	Type I Error Rate
			AMH	BD
1:1	Low	Easy	4.45–5.92	5.21–5.42
2:1	Low	Easy	4.90–5.20	4.14–5.16
5:1	Low	Easy	4.61–6.01	4.01–5.79
1:1	Low	Medium	4.62–6.03	4.08–5.36
2:1	Low	Medium	4.55–5.23	4.29–5.14
5:1	Low	Medium	4.77–6.04	4.68–5.36
1:1	Low	Hard	4.65–6.08	4.74–5.19
2:1	Low	Hard	4.20–5.65	4.43–5.20
5:1	Low	Hard	4.83–6.10	4.10–5.74
1:1	High	Easy	4.67–6.05	4.92–5.30
2:1	High	Easy	4.85–5.20	4.40–5.73
5:1	High	Easy	4.53–5.57	4.82–5.13
1:1	High	Medium	4.55–5.68	4.60–5.76
2:1	High	Medium	4.43–5.11	4.00–5.94
5:1	High	Medium	4.48–6.00	4.33–5.36
1:1	High	Hard	4.35–5.53	4.70–5.08
2:1	High	Hard	4.15–5.19	4.28–5.65
5:1	High	Hard	4.72–5.35	4.40–5.19

Table 4. Detection rates from unequal ability distributions for no DIF items (type I error rates).

R:F Ratio	Discrimination	Difficulty	DIF Detection Rate (Type I Error)
			AMH	BD
1:1	Low	Easy	4.09–6.04	4.31–5.96
2:1	Low	Easy	4.32–5.40	4.73–5.20
5:1	Low	Easy	4.08–5.44	4.18–5.39
1:1	Low	Medium	4.48–5.56	4.32–5.92
2:1	Low	Medium	4.19–5.71	4.52–5.19
5:1	Low	Medium	4.30–5.16	4.78–5.24
1:1	Low	Hard	4.40–5.54	4.29–5.88
2:1	Low	Hard	4.32–5.56	4.49–5.46
5:1	Low	Hard	4.29–5.25	4.62–5.16
1:1	High	Easy	4.64–6.06	4.72–6.14
2:1	High	Easy	4.22–6.09	4.69–5.38
5:1	High	Easy	4.22–5.28	4.60–5.31
1:1	High	Medium	4.32–5.58	4.46–5.80
2:1	High	Medium	4.39–5.72	4.55–5.34
5:1	High	Medium	4.16–5.22	4.72–5.12
1:1	High	Hard	4.68–5.46	4.88–5.96
2:1	High	Hard	4.42–5.52	4.59–5.53
5:1	High	Hard	4.18–5.35	4.65–5.26

Upon a closer examination of the Type I errors for no DIF items under equal ability distributions, several nuanced observations emerge. For items with low discrimination and easy difficulty, the AMH procedure’s Type I error rates spanned from 4.45% to 6.01% across different R:F ratios, while the BD procedure’s rates ranged from 4.01% to 5.79%. Notably, in the 5:1 R:F ratio condition, the AMH procedure exhibited a slightly wider range. When considering items with low discrimination and medium difficulty, the AMH procedure’s rates varied between 4.55% and 6.04%, whereas the BD procedure’s rates were more constrained, lying between 4.08% and 5.36%. Both procedures showed comparable variability in this scenario. For items with low discrimination and hard difficulty, the AMH procedure’s rates were between 4.20% and 6.10%, while the BD procedure’s rates hovered between 4.10% and 5.74%. Here, the AMH procedure demonstrated a marginally wider range, especially evident in the 5:1 R:F ratio condition. When the discrimination was high, the AMH procedure’s Type I error rates generally spanned from 4.15% to 6.06% across all difficulty levels. In contrast, the BD procedure’s rates were slightly narrower, ranging from 4.00% to 5.94%. The AMH procedure showed a broader range, particularly in the "Easy" difficulty level with a 1:1 R:F ratio.

Several patterns emerge when examining the Type I errors for no DIF items under unequal ability distributions. For items with low discrimination and easy difficulty, the AMH procedure’s Type I error rates spanned from 4.08% to 6.04% across different R:F ratios, while the BD procedure’s rates ranged from 4.18% to 5.96%. Notably, in the 1:1 R:F ratio condition, the AMH procedure exhibited a slightly wider range. When considering items with low discrimination and medium difficulty, the AMH procedure’s rates varied between 4.19% and 5.56%, whereas the BD procedure’s rates were more constrained, lying between 4.32% and 5.92%. Both procedures showed comparable variability in this scenario. For items with low discrimination and hard difficulty, the AMH procedure’s rates were between 4.29% and 5.54%, while the BD procedure’s rates hovered between 4.29% and 5.88%. Here, the BD procedure demonstrated a marginally wider range, especially evident in the 1:1 R:F ratio condition. When the discrimination was high, the AMH procedure’s Type I error rates generally spanned from 4.16% to 6.06% across all difficulty levels. In contrast, the BD procedure’s rates were slightly narrower, ranging from 4.46% to 6.14%. The BD procedure showed a broader range, particularly in the "Easy" difficulty level with a 1:1 R:F ratio.

In summary, the Type I error rates for equal and unequal ability distributions indicate that both procedures, AMH and BD, maintained Type I error rates that were consistently at the nominal level of 0.05.

The detection rate of non-uniform DIF items, or the statistical power, refers to the percentage of simulations in which the AMH and the BD procedures successfully detected non-uniform DIF in the studied items (DIF items), which are shown in Tables 5 and 6. Both reflect percentages as a function of sample size ratios (RF), item discrimination levels (AL), and item difficulty levels (BL), but the two tables contrast by the consideration of equal and unequal ability distributions.

Table 5. Non-uniform DIF detection rates from equal ability distributions.

R:F Ratio	Discrimination	Difficulty	Statistical Power
			AMH	BD
1:1	Low	Easy	95.51–97.87	47.28–54.91
2:1	Low	Easy	89.35–91.34	34.70–42.17
5:1	Low	Easy	69.96–70.27	22.28–25.49
1:1	Low	Medium	47.85–49.95	43.68–52.39
2:1	Low	Medium	37.40–44.79	29.76–37.77
5:1	Low	Medium	25.86–27.93	18.27–21.82
1:1	Low	Hard	88.14–89.65	18.53–22.26
2:1	Low	Hard	73.03–75.16	14.05–16.75
5:1	Low	Hard	46.42–48.88	9.74–10.85
1:1	High	Easy	100.00–100.00	99.81–99.98
2:1	High	Easy	100.00–100.00	98.96–99.16
5:1	High	Easy	99.77–100.00	90.51–91.87
1:1	High	Medium	99.43–99.92	99.91–99.95
2:1	High	Medium	96.27–98.32	98.97–99.22
5:1	High	Medium	84.07–93.64	86.64–87.58
1:1	High	Hard	99.95–99.99	39.89–44.43
2:1	High	Hard	99.22–99.47	28.75–29.69
5:1	High	Hard	89.13–90.07	16.68–19.44

Table 6. Non-uniform DIF detection rates from unequal ability distributions.

R:F Ratio	Discrimination	Difficulty	Statistical power
			AMH	BD
1:1	Low	Easy	93.03–94.16	46.27–47.47
2:1	Low	Easy	85.37–85.93	31.48–32.15
5:1	Low	Easy	64.89–68.29	18.30–21.95
1:1	Low	Medium	48.59–48.90	35.92–36.26
2:1	Low	Medium	34.82–34.91	24.72–25.11
5:1	Low	Medium	22.85–22.95	14.52–14.63
1:1	Low	Hard	88.90–89.40	11.72–11.86
2:1	Low	Hard	72.76–73.06	9.56–9.86
5:1	Low	Hard	46.63–47.14	7.41–7.72
1:1	High	Easy	100.00–100.00	99.95–99.98
2:1	High	Easy	100.00–100.00	99.23–99.39
5:1	High	Easy	99.87–99.90	89.00–89.31
1:1	High	Medium	99.58–99.63	99.39–99.45
2:1	High	Medium	95.15–95.33	95.63–95.78
5:1	High	Medium	74.62–74.83	73.85–73.98
1:1	High	Hard	99.96–99.99	16.04–16.16
2:1	High	Hard	99.20–99.27	11.58–11.87
5:1	High	Hard	89.44–89.73	7.51–7.80

For reference and focal groups with equal ability distributions, detection rates for studied items ranged from 25.86% to 100.00% for the AMH procedure and 9.74% to 99.98% for the BD procedure. The variability of detection rates was higher for the BD procedure than for the AMH procedure. The three highest detection rates for the AMH procedure occurred with items containing high ALs and easy BLs, while the three lowest detection rates occurred with items containing low ALs and medium BLs. A smaller number of combinations with high detection rates were observed for the BD procedure. The three highest detection rates were observed for the BD procedure when items contained high ALs and easy or medium BLs. The three lowest detection rates were for low ALs, 2:1 or 5:1 RFs, and medium or hard BLs. Regardless of the RFs, the AMH procedure had a lower potential of detecting non-uniform DIF when the AL was low and the BL was medium. Similarly, the BD procedure had a lower potential of detecting non-uniform DIF when the AL was low, and the BL was hard.

Similar behavior was observed with the two procedures when the ability distributions were unequal. Detection rates ranged between 22.85% to 100.00% for the AMH procedure and 7.41% to 99.98% for the BD procedure, with more variability in the BD procedure’s detection rates. The three highest detection rates for the AMH procedure were found in items containing high ALs and easy BLs, even though some high detection rates were observed even when the BL increased. The lowest detection rates for the AMH procedure were found when the AL was low and the BL was medium, regardless of the RF ratio. For the BD procedure, the highest detection rates were observed for items containing high ALs and easy or medium BLs. The lowest detection rates occurred with hard BLs and large RFs (7.41% and 7.51%). The results from both tables suggest that considering equal or unequal ability distributions does not significantly affect non-uniform DIF detection rates. Instead, the key factors include the size of the RF and item characteristics.

To further understand the behavior and dependence that the AL, BL, and RF have on non-uniform DIF detection rates, four full-effect logistic regression models were created to predict the likelihood of detecting item non-uniform DIF. They are detecting non-uniform DIF for the AMH procedure with equal ability distributions (Model 4a), detecting non-uniform DIF for the BD procedure with equal ability distributions (Model 4b), detecting non-uniform DIF for the AMH procedure with unequal ability distributions (Model 4c), and detecting non-uniform DIF for the BD procedure with unequal ability distributions (Model 4d). Table 7 shows the estimates and standard errors from the four models and the AIC, BIC, c-statistic, and CCR to represent the model diagnostics.

Table 7. AMH & BD Full-Effects logistic Models.

	Equal				Unequal
	AMH (4a)		BD (4b)		AMH (4c)		BD (4d)
Estimates	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$
Intercept (${\beta }_0$)	2.89		−0.19		1.32		−0.10
Reference-Focal Ratio (RF)
RF = 2:1	−1.26***	0.06	−0.52***	0.03	−0.58***	0.09	−0.68***	0.08
RF = 5:1	−2.62***	0.06	−1.14***	0.03	−1.77***	0.33	−1.17**	0.38
Discrimination Level (AL)
AL = high	15.32	89.84	8.63***	0.71	17.00	95.12	8.21***	0.58
Difficulty Level (BL)
BL = medium	−5.86***	0.07	−0.15**	0.03	−3.40***	0.09	−0.46***	0.07
BL = hard	−0.87***	0.17	−1.34***	0.14	1.08***	0.09	−1.91***	0.07
Two-Way Interactions
RF = 2 $\times$ AL = high	1.26	145.10	−3.22***	0.72	−10.61	95.12	−2.57***	0.59
RF = 5 $\times$ AL = high	−11.16	89.84	−5.12***	0.72	−13.03	95.12	−4.84***	0.69
RF = 2 $\times$ BL = medium	1.17***	0.14	−0.08	0.07	0.12	0.13	0.13	0.09
RF = 2 $\times$ BL = hard	0.09	0.17	0.19	0.14	−0.63***	0.10	0.48***	0.09
RF = 5 $\times$ BL = medium	2.60***	0.10	−0.10	0.06	1.14***	0.34	−0.34	0.39
RF = 5 $\times$ BL = hard	0.30	0.17	0.40**	0.14	−0.70*	0.33	0.69	0.39
AL = high $\times$ BL = medium	−15.38	89.84	−0.95	0.82	−15.29	95.12	−2.55	0.59
AL = high $\times$ BL = hard	−9.78	89.84	−7.34***	0.72	1.08	134.50	−7.84	0.58
Three-Way Interactions
RF = 2 $\times$ AL = high $\times$ BL = medium	−0.94	145.10	1.25	0.84	10.16	95.12	1.11	0.61
RF = 2 $\times$ AL = high $\times$ BL = hard	−1.91	145.10	2.91	0.74	0.07	134.50	2.41	0.60
RF = 5 $\times$ AL = high $\times$ BL = medium	11.56	89.84	0.90	0.83	11.95	95.12	2.00**	0.71
RF = 5 $\times$ AL = high $\times$ BL = hard	8.18	89.84	4.67***	0.73	−0.13	134.50	4.49***	0.70
AIC	69,363.72		82,660.50		86,121.56		91,351.22
BIC	69,537.09		82,833.87		86,299.44		91,529.10
c	0.935		0.895		0.938		0.924
CCR	85.88%		82.01%		86.61%		88.08%

Note.

* $p<5\mathrm{\%},$ **$p<1\mathrm{\%},$ ***$p<0.1\mathrm{\%}.$

It is important to note that the current study focused on Models 4a and 4c. Models 4b and 4d were created to compare the significance found in these models to those of Models 4a and 4c, thus identifying concordant and discordant predictor behavior when detecting non-uniform DIF using the AMH versus the BD procedures.

Table A1 through Table A4 of the Appendix shows model-building, main-effect models to demonstrate each predictor’s inclusion in the model and to determine whether a predictor’s inclusion affects the significance or change in its effect.

The AIC and BIC for Model 4a were 69,363.72 and 69,537.09, respectively, which are substantially lower than the AIC and BIC values from the main-effects models in Table A1. The c-statistic and CCR of 0.935 and 85.88%, respectively, suggest that this model possesses solid predictive power. The RF was found to have a powerfully negative effect on the detection of non-uniform DIF. Non-uniform DIF items from RFs of 2:1 and 5:1 had lower likelihoods of being detected than non-uniform DIF items from an RF of 1:1, which decreased the log of detection odds by 1.26 and 2.62 logits, respectively. The RF predictor behavior was found to be concordant with that of the RF predictor from Model 4b. The AL was found to be a poor predictor when estimating non-uniform DIF items and strongly discordant to the AL predictor’s behavior in Model 4b. Significant negative effects were present with the BL predictor. These effects suggest that non-uniform DIF items with hard BLs have lower detection rates than items with easy BLs when using the AMH procedure ($\widehat{\beta }=-0.87,p_v<0.1\text{\%}$). However, items with medium BLs have even lower detection rates ($\widehat{\beta }=-5.86,p_v<0.1\text{\%}$). Two-way and three-way interactions of these predictors were found to be insignificant. Significant positive effects were present only for the interaction between the RF and BL, particularly for medium BLs items involving 2:1 RF ($\widehat{\beta }=1.17,p_v<0.1\text{\%}$) or 5:1 RFs ($\widehat{\beta }=2.60,p_v<0.1\text{\%}$). These significant interactions were discordant to the behaviors and significant interactions found in Model 4b with the RF and AL predictors (negative effects with the 2:1 and 5:1 RF levels with the high AL), RF and BL predictors (positive effect with the 5:1 RF with the hard BL level), the AL and BL predictors (negative effect with the high AL and hard BL), and all three predictors (positive effect with the 5:1 RF, high AL, and hard BL).

Model 4c had an AIC of 86,121.56 and a BIC of 86,299.44, which are lower than the AIC and BIC values of the main-effects models of Table A3. The c-statistic and CCR for Model 4c were 0.938 and 86.61%, respectively, which was just slightly higher than those calculated in Model 4d. The RF predictor was found to have a strong negative effect on non-uniform DIF detection. Items created from 2:1 or 5:1 RFs decreased DIF detection rates by −0.58 and −1.77 logits. It is interesting to recognize that the behaviors and significances observed with the RF predictor were similar in Models 4c and 4d, while the standard errors differed. One plausible reason could be the differences in the ability distributions. These results were similar to those in Models 4c and 4d, where the effects were negative, and the standard errors were equivalent. Discrimination was not statistically significant in Model 4c, which was the opposite of the effects observed in Model 4d. Strong significant effects were present with the BL on detection rates in Models 4c and 4d. Items with medium BLs decreased detection rates by 3.40 logits, but hard items tended to increase detection rates by 1.08 logits. These results were discordant with what was observed in Model 4d, in which medium and hard items exhibited a significant negative effect on the likelihood of detecting non-uniform DIF.

The similarities found in Models 4a and 4c suggest that some factors contribute to non-uniform DIF detection rates regardless of differences in ability level distributions. As the ratio of the reference and focal groups’ sizes increase, the likelihood of detecting non-uniform DIF with the AMH test may decrease. Results showed that groups with a 2:1 ratio item significantly decreased DIF detection between 0.58 logits and 1.26 logits. In comparison, groups with a 5:1 ratio item significantly decreased DIF detection between 1.77 to 2.62 logits. Medium items also significantly decreased DIF detection with the AMH procedure between 3.40 and 5.86 logits. However, the interaction of the RF and BL factors had a significant positive effect on non-uniform DIF detection as they increased the likelihood between 1.14 and 2.60 logits.

Real application

In the context of this study, "ability" denotes the students’ proficiency or competence in the subject matter, as gauged by their test scores from the "Introduction to Statistics" course. This construct of ability is central to the AMH procedure, emphasizing its importance in categorizing students into distinct ability levels to facilitate a DIF analysis. For the DIF analysis, students with test scores less than 70% (equivalent to a grade below C) were classified as having "Low Ability," while those who achieved scores of 70% or more (equivalent to a grade of C and above) were designated as having "High Ability."

Table 8 cross-classified students according to their gender, major, and ability level. Around 41.44% of male students’ ability level was low, while 58.56% was high. For female students, the low and high ability percentage was 32.51% and 67.49%, respectively. Finally, 31.90% of STEM students had a low ability level, and 68.10% had a high ability level. For non-STEM students, 39.45% had low ability levels, and 60.55% had high ability levels.

Table 8. Ability level by gender and major.

Groups		Ability Level
		Low	High
By Gender	Male	41.44%	58.56%
	Female	32.51%	67.49%
By Major	STEM	31.90%	68.10%
	Non-STEM	39.45%	60.55%

Figure 1 shows the percentages of correct answers rate of study items, according to gender (study items are identified in Table B1, Appendix B). Around 64.4% of male and 59.51% of female students answered Item 1 correctly. For Item 2, 50% of male and 50.38% of female students answered it correctly. The correct answer rate for Item 3 was 83.85% for males and 73.95% for females. Around 86.19% of male and 79.85% of female students answered Item 4 correctly. Finally, the percentage of correct answers rate for item 5 was 63.42% for males and 76.81% for females.

Figure 1. Percentages of correct answers rate of study items by gender.

Figure 2. Percentages of correct answers rate of study items by major.

Figure 2 shows the percentage of correct answers rate of study items according to students’ major. Around 83.05% of STEM students and 51.3% of non-STEM students answered Item 1 correctly. For Item 2, 60.92% of STEM students and 44.8% of non-STEM students answered it correctly. The correct answer rate for Item 3 was 89.08% for STEM and 73.7% for non-STEM students. Regarding Item 4, 80.17% of STEM and 84.39% of non-STEM students answered Item 4 correctly. Finally, 61.21% of STEM and 74.71% of non-STEM students answered Item 5 correctly.

Table 9 shows parameter estimation values for item guessing, difficulty, and discrimination parameters. The parameter estimates are obtained by maximizing the marginal likelihood using the quasi-Newton algorithm in the IRT procedure in SAS. The guessing parameters ranged between 0.129 and 0.171, suggesting that respondents with low ability may answer the questions correctly. Moreover, in terms of difficulty level, it ranged between −1.293 and 0.497. This indicates that Item 4 is the easiest and Item 2 is the hardest. The discrimination parameters ranged between 0.773 and 1.667. In general, if an item has the highest discrimination value, it may discriminate against examinees more accurately and clearly. Based on Table 9, Item 2 has the lowest discrimination value, whereas Item 1 has the highest value.

Table 9. Items parameter estimates.

Items	Parameter	Estimate	se
1	Guessing (c)	.129	.062
	Difficulty (b)	−0.229	.138
	Discrimination (a)	1.667	.338
2	Guessing (c)	.148	.074
	Difficulty (b)	.497	.279
	Discrimination (a)	.773	.189
3	Guessing (c)	.171	.089
	Difficulty (b)	−1.284	.294
	Discrimination (a)	.999	.185
4	Guessing (c)	.138	.072
	Difficulty (b)	−1.293	.180
	Discrimination (a)	1.510	.261
5	Guessing (c)	.143	.075
	Difficulty (b)	−0.789	.238
	Discrimination (a)	.948	.170

AMH and BD procedures are used to test the null hypothesis of no DIF detection (An Item is not biased). Table 10 shows whether DIF is detected in different scenarios or not. The Type of DIF is decided based on the graphical illustration suggested by Amaechi and Onah (2020) and Walker (2011), where non-parallel ICCs refer to non-uniform DIF, as given in Figure C1 in Appendix C.

Table 10. DIF detection for different scenarios.

Items	Grouping Variable	Type of DIF	Detection Procedure	Chi-square	p-value
1	Gender	Non-Uniform	AMH	2.768	0.096
			BD	7.322	0.007
	Major	Uniform	AMH	98.823	<0.001
			BD	5.408	0.020
2	Gender	Non-Uniform	AMH	0.022	0.882
			BD	16.692	<0.001
	Major	Non-Uniform	AMH	22.260	<0.001
			BD	7.119	0.008
3	Gender	Non-Uniform	AMH	15.812	<0.001
			BD	17.275	<0.001
	Major	Uniform	AMH	32.516	<0.001
			BD	0.291	0.590
4	Gender	Non-Uniform	AMH	8.137	0.004
			BD	0.283	0.595
	Major	Non-Uniform	AMH	3.290	0.070
			BD	1.981	0.159
5	Gender	Non-Uniform	AMH	20.305	<0.001
			BD	3.791	0.052
	Major	Non-Uniform	AMH	22.253	<0.001
			BD	0.152	0.697

Item 1 was found to be a gender-bias item when using the BD procedure (Chi-square = 7.322, p-value = 0.007) and a major-bias item when using the AMH procedure (Chi-square = 98.823, p-value < 0.001) and the BD procedure (Chi-square = 5.408, p-value = 0.020). Concerning Item 2, it was found to be a gender-bias item when using the BD procedure (Chi-square = 16.692, p-value < 0.001) and a major-bias item when using both the AMH procedure (Chi-square = 22.260, p-value < 0.001) and the BD procedures (Chi-square = 7.119, p-value = 0.008). Item 3 was a gender-bias item when using both the AMH procedure (Chi-square = 15.812, p-value < 0.001) and the BD procedures (Chi-square = 17.275, p-value < 0.001), and a major-bias when using the AMH procedure (Chi-square = 32.516, p-value < 0.001). Item 4 was found to be gender-biased (Chi-square = 8.137, p-value = 0.004) when the AMH procedure was applied. Finally, Item 5 was found to be a gender bias (Chi-square = 20.305, p-value < 0.001), and a major-bias item (Chi-square = 22.253, p-value < 0.001), when the AMH procedure was applied.

Discussion

This study aimed to determine whether changes in sample size ratios of the reference and focal groups, item discrimination, and item difficulty affected how often the AMH and BD procedures detected non-uniform DIF. It also examined the factors that significantly affected the AMH procedure’s detection rate and whether particular combinations of these factors yielded higher detection rates with the AMH procedure compared to the BD procedure.

With equal ability distributions, the results suggest that the detection rate of non-uniform DIF using the AMH procedure is most affected by items with high discrimination, followed by items with easy item difficulty, and items answered by reference to a focal group of equal sizes. This was based on comparing the six most considerable detection rates in Table 1. Using the AMH procedure, items with high discrimination tend to have an 84% chance of being detected, 70% or more for items with easy difficulty, and 50% or more for items with equal reference and focal group ratios. Items associated with all three characteristics have the highest chance of being detected by this procedure. The order of these factors may be trivial since item characteristics have been shown to play an essential part in detecting non-uniform DIF using several other DIF procedures.

Differences in reference and focal group ratios have a significant negative effect when modeling non-uniform DIF detection rates. The significance becomes greater as the magnitude of the difference increases. Item difficulty also has a significantly negative effect on non-uniform DIF detection rates with the AMH procedure. Items with medium or hard difficulty exhibit lower chances of being detected, with medium-level items possessing the lowest detection chance. The negative effect for medium items slightly offsets higher group ratios.

For the unequal ability distribution case, the results are slightly similar. Items with high discrimination, followed by items with easy item difficulty, and then items answered by reference and a focal group of equal sizes offer the highest detection rates for non-uniform DIF with the AMH procedure. Items with high discrimination tend to have about 75% chance or more of being detected, 68% or more for items with easy difficulty, and about a 49% chance or more for items with equal reference and focal group ratios. These percentages are similar to those of the equivalent case. So, non-uniform DIF detection is believed to be insensitive to unequal ability levels between the reference and focal groups.

Group ratio and item difficulty still have significant effects on non-uniform DIF detection rates. Group ratio has a dominant-negative effect, and items involving higher ratios have a more substantial negative effect. Regarding item difficulty, medium items harm the detection rates of uniform DIF compared to easy items, but hard items have a positive impact. This behavior is in contrast to what was observed with the equivalent ability distribution case. It is also discordant with the behavior found that would be observed using the BD procedure, in which item difficulty has an entirely negative effect. Table 11 provides a summary of the comparison between the AMH and BD methods for non-uniform DIF detection across various item discrimination, item difficulty, and group ratio combinations.

Table 11. AMH vs. BD performance in non-uniform DIF detection across item characteristics.

Item discrimination	Item difficulty	Group ratio	Better performing method
High	Easy	Equal	AMH
High	Easy	Unequal	AMH
High	Medium	Equal	AMH
High	Medium	Unequal	BD
High	Hard	Equal	AMH
High	Hard	Unequal	AMH
Low	Easy	Equal	BD
Low	Easy	Unequal	BD
Low	Medium	Equal	BD
Low	Medium	Unequal	BD
Low	Hard	Equal	BD
Low	Hard	Unequal	BD

The results suggest that the effects of item difficulty and group ratios are similar to those observed using the BD procedure. However, the effects observed for item discrimination would differ. One possible reason could be that the test statistic formulas involved with the two methods are sample-size dependent. The AMH procedure is a particular situation of the MH procedure, and like other IRT-based methods, it is affected by item difficulty. Regarding sample size, it expresses how decreases in group sizes significantly affect the MH's ability to detect DIF-affected items.

The simulation study indicates that the AMH procedure detects non-uniform DIF, which is the best for items with easy difficulty levels or high discrimination. Test analysts should expect at least a 2.36:1 chance of successfully detecting non-uniform DIF when either property is present in an item.

Compared to the BD procedure, the AMH procedure has a higher chance of detecting non-uniform DIF items except when items contain high discrimination and medium difficulty. The AMH procedure yields the most substantial potential to detect non-uniform DIF items for easy difficulty or high discrimination and the weakest detection rates for low-discriminating, medium-difficulty items.

Results from the application study showed that out of ten scenarios, the AMH procedure detected DIF in seven scenarios. On the other hand, the BD procedure could detect DIF in only five scenarios. In agreement with the simulation study, results from the application study showed that the AMH procedure was more effective in DIF detection for both uniform and non-uniform cases compared to the BD procedure.

In response to the research questions, the findings reveal that when there is a significant difference in sample sizes between the groups studied, the performance of the AMH procedure in detecting non-uniform DIF is affected. The simulation study observed that the detection rate of non-uniform DIF was negatively impacted as the magnitude of the difference in sample sizes between the reference and focal groups increased. The application study also supported this finding, where the AMH procedure was more effective in DIF detection for both uniform and non-uniform cases when the groups had relatively similar sizes.

Moreover, the effectiveness of the AMH procedure was evaluated in different scenarios based on the sample size ratios between the reference and focal groups. The simulation study indicated that the AMH procedure was particularly effective in detecting non-uniform DIF for items with easy difficulty levels or high discrimination. The application study further supported these findings, as the AMH procedure successfully detected DIF in seven out of ten scenarios, while the BD procedure detected DIF in only five scenarios. These results suggest that the AMH procedure is generally more effective in detecting non-uniform DIF items, mainly when there are no significant differences in the sample sizes between the groups studied.

Based on the simulation and application studies, the AMH procedure generally had a higher chance of detecting non-uniform DIF items than the BD procedure, except for items with high discrimination and medium difficulty. The most favorable conditions for the AMH procedure were when items had easy difficulty or high discrimination. In contrast, the weakest detection rates occurred for low-discriminating, medium-difficulty items. These findings provide valuable insights for test analysts and researchers when choosing the appropriate method for detecting non-uniform DIF. The AMH procedure demonstrates better performance under specific conditions than the BD procedure.

There are exciting limitations worth noting regarding this research. First, in each simulation, the detection (power) rates were carried out with three study items. This can be extended using more study items containing non-uniform using AMH and BD procedures. Moreover, recall that the application study investigated 1,040 collegiate students from a national university, slightly smaller than the 1,200 to 2,000 observations used in the simulation study. Researchers may extend this research by increasing the ratio between the reference and focal groups, then examining how detectability will be affected by such changes.

Furthermore, this research used students’ test scores to categorize them into low and high ability levels, which is another limitation. Several other factors can signify or measure students’ ability, such as language and motivation to learn. A final limitation involved the types of DIF procedures performed. While the current study tested the effects of several factors on non-uniform DIF detection rate using AMH and BD procedures, additional procedures can be considered. Comparing AMH and BD procedures to other approaches such as IRT, logistic regression, and simultaneous item bias test (SIBTEST) procedures may interest some education analysts.

Conclusion

This study underscores the conditions under which the AMH procedure excels in detecting non-uniform DIF items. Specifically, the AMH procedure outperforms the BD procedure when items possess high discrimination or are of easy difficulty and when the ratios between reference and focal groups are either equal or not markedly different. However, a critical observation is that the AMH and BD procedures falter when confronted with low discrimination and medium difficulty.

Test analysts and researchers need to be cognizant of these conditions when opting for a DIF detection method. The efficacy of each method is contingent upon these factors; in situations where the AMH and BD methods show suboptimal performance, exploring alternative techniques or a fusion of methods might be a strategic move to enhance DIF detection precision.

While the study sheds light on the comparative performance of the AMH and BD procedures in varied scenarios, it is not without limitations, notably the number of items studied and the criteria adopted to gauge students’ abilities. Nevertheless, the findings serve as a beacon for test analysts and researchers in their quest for the most fitting method for non-uniform DIF detection. A deeper dive into diverse DIF procedures, exploration of varied conditions, and addressing the present study’s limitations can pave the way for more robust DIF detection methodologies in educational testing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Appendix A. 

Main effects logistic models

Table A1. AMH main effects logistic Models: Equal ability distributions.

Estimates	Model 1		Model 2		Model 3
	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$
Intercept ($\alpha$)	0.02		0.11		2.37
Reference-Focal Ratio
$R{F}_2$	0.51***	0.01	0.50***	0.01	−0.68***	0.03
$R{F}_5$	−0.41***	0.02	−0.46***	0.02	−1.91***	0.03
Discrimination Level
$A{L}_H$			−0.16***	0.01	2.13***	0.04
Difficulty Level
$B{L}_M$					−6.49***	0.04
$B{L}_T$					−0.73***	0.02
AIC	152,078.76		151,921.62		72,911.56
BIC	152,107.65		151,960.15		72,969.35
c	0.599		0.606		0.934
CCR	57.69%		55.19%		85.88%

Note.

* $p<5\mathrm{\%},$

** $p<1\mathrm{\%},$

*** $p<0.1\mathrm{\%}.$

Table A2. BD main effects logistic Models: Equal ability distributions.

Estimates	Model 1		Model 2		Model 3
	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$
Intercept ($\alpha$)	0.02		− 0.16		2.37
Reference-Focal Ratio
$R{F}_2$	−0.83***	0.02	−0.88***	0.02	−0.35***	0.02
$R{F}_5$	−1.81***	0.02	−1.50***	0.02	−1.05***	0.02
Discrimination Level
$A{L}_H$			2.91***	0.02	3.16***	0.02
Difficulty Level
$B{L}_M$					0.01	0.02
$B{L}_T$					−2.54***	0.03
AIC	142,446.36		103,656.00		90,064.18
BIC	142,475.25		103,694.52		90,121.97
c	0.681		0.854		0.889
CCR	64.85%		79.59%		80.26%

Note.

* $p<5\mathrm{\%},$

** $p<1\mathrm{\%},$

*** $p<0.1\mathrm{\%}.$

Table A3. AMH main effects logistic Models: Unequal ability distributions.

Estimates	Model 1		Model 2		Model 3
	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$
Intercept ($\alpha$)	1.18***		0.53***		0.82***
Reference-Focal Ratio
$R{F}_2$	−0.46***	0.01	−0.52***	0.01	−0.76***	0.02
$R{F}_5$	−1.07***	0.01	−1.23***	0.01	−1.79***	0.02
Discrimination Level
$A{L}_H$			1.65***	0.01	2.42***	0.02
Difficulty Level
$B{L}_M$					−2.04***	0.02
$B{L}_T$					1.86***	0.02
AIC	182,971.03		173,816.81		89,748.51
BIC	183,000.68		173,856.34		89,807.81
c	0.617		0.741		0.888
CCR	65.46%		70.62%		80.26%

Note.

* $p<5\mathrm{\%},$

** $p<1\mathrm{\%},$

*** $p<0.1\mathrm{\%}.$

Table A4. BD main effects logistic Models: Unequal ability distributions.

Estimates	Model 1		Model 2		Model 3
	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$	$\beta$	$s{e}_{\beta }$
Intercept ($\alpha$)	0.13***		−0.65***		0.50***
Reference-Focal Ratio
$R{F}_2$	−0.27***	0.01	−0.32***	0.01	−0.51***	0.02
$R{F}_5$	−0.75***	0.01	−0.87***	0.01	−1.40***	0.02
Discrimination Level
$A{L}_H$			1.59***	0.01	2.58***	0.02
Difficulty Level
$B{L}_M$					−0.70***	0.01
$B{L}_T$					−4.34***	0.02
AIC	197,703.77		162,997.73		103,168.52
BIC	197,733.42		163,037.26		103,227.81
c	0.582		0.724		0.903
CCR	53.27%		68.17%		83.63%

Note.

* $p<5\mathrm{\%},$

** $p<1\mathrm{\%},$

*** $p<0.1\mathrm{\%}.$

Appendix B. 

Study items

Table B1. Study items included in real application.

Items	Description
1	The weights of books on one shelf in library are mounded shape with a mean of 800 grams and a standard deviation of 40 grams. The percentage of books weighing between 720 grams and 880 grams is about:
2	The third quartile of the standard normal distribution is equal to:
3	Given a population proportion of p = 0 .6 and a sample size of n = 150, then the standard deviation of the sample proportion is:
4	It is known that 85% of all the orders it receives from its customers are delivered on time, then probability that the proportion of orders in which a random sample of 100 are delivered on time will be less than 0.87 is:
5	If P(A) = 0.55, P(B) = 0.45, and P(A and B) = 0.33, then P(A|B) is:

Appendix C. 

Item characteristic curves (ICCs) for study items

Figure C1. Item characteristic curves (ICCs) for study item.