Adjusting for guessing and applying a statistical test to the disaggregation of value-added learning scores

Abstract

In 2016, Walstad and Wagner developed a procedure to split pre-test and post-test responses into four learning types: positive, negative, retained, and zero learning. This disaggregation is not only useful in academic studies; but also provides valuable insight to the practitioner: an instructor would take different mitigating actions in response to zero versus negative learning. However, the original disaggregation is sensitive to student guessing. This article extends the original work by accounting for guessing and provides adjusted estimators using the existing disaggregated values. Further, Monte Carlo simulations of the adjusted learning type estimates are provided. Under certain assumptions, an instructor can determine if a difference in positive (or negative) learning is the result of a true change in learning or “white noise.”

Keywords:

• difference score
• learning disaggregation
• guessing
• simulation

JEL CODES:

• A20
• A22
• A23
• C63

Acknowledgment

The authors thank Bill Goffe, Matt Rousu, Bill Walstad, Dustin White, Mark Wohar, and the four anonymous referees for helpful suggestions on this project. They further thank Halli Tripe for editorial suggestions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 While guessing is addressed in Salemi and Tauchen (1980), most studies concerned with inaccuracies associated with estimating learning relate to either sampling error (e.g., Bowles and Jones 2003; Koedel, Mihaly, and Rockoff 2015, 183–85) or data loss (Becker and Powers 2001; Becker and Walstad 1990).

2 We use hats (e.g., $\widehat{\text{pl}}$) throughout this article to emphasize when we are discussing an estimate. Thus, $\mu$ is a true underlying parameter while $\widehat{\mu }$ is an estimate of $\mu$ using the data.

3 This holds true as long as true positive learning is greater than true negative learning (forgetting). The proportion of students guessing on the pre-test is $1-\mu$, while the proportion guessing on the post-test (assuming no true negative learning) is $1-\mu -\gamma$.

4 It is possible that the probability of guessing correctly is different on the pre-test and post-test assuming success is correlated to student ability. However, a usable modification to our assumed probability ($1/n$) is not readily available. Further, the primary estimation technique available in the item response theory literature is ill-suited to detect such a difference; the pseudo-guessing parameter in the three-parameter logistic model is the probability that a very low ability student responds correctly nonetheless. By design, the model is attempting to remove ability as a factor.

5 However, the bounds of $\gamma$ depend on $\mu$. $\gamma$ has an upper constraint at $1-\mu$ .

6 However, like $E[\widehat{\text{flow}}],$$E[\widehat{\text{zl}}]$ can be treated as an ordinal or count variable as it depends only on the probability of guessing correctly.

7 $\Delta E[\widehat{\text{pl}}]/\Delta \mu$ shows the resulting change in $E[\widehat{\text{pl}}]$ with a one unit change in $\mu$. Therefore, the resulting change in $\Delta E[\widehat{\text{pl}}]$ from a $0.1$ unit change in ? can be calculated as follows: $0.1\times ((1-4)/4^2).$

8 For the reader’s convenience we have included equation 3(3) $$\begin{array}{c}\widehat{\mu }=\frac{\widehat{\text{nl}}+\widehat{\text{rl}}-1}{n-1}+\widehat{\text{nl}}+\widehat{\text{rl}}\\ \widehat{\gamma }=\frac{n(\widehat{\text{nl}}+\widehat{\text{pl}}n+\widehat{\text{rl}}-1)}{{(n-1)}^2}\\ \widehat{\alpha }=\frac{n(\widehat{\text{nl}}n+\widehat{\text{pl}}+\widehat{\text{rl}}-1)}{{(n-1)}^2}\end{array}$$(3) expressed in terms of the probability of guessing correctly ($\widehat{p}$) instead of the number of answer options ($n$). Equation 3(3) $$\begin{array}{c}\widehat{\mu }=\frac{\widehat{\text{nl}}+\widehat{\text{rl}}-1}{n-1}+\widehat{\text{nl}}+\widehat{\text{rl}}\\ \widehat{\gamma }=\frac{n(\widehat{\text{nl}}+\widehat{\text{pl}}n+\widehat{\text{rl}}-1)}{{(n-1)}^2}\\ \widehat{\alpha }=\frac{n(\widehat{\text{nl}}n+\widehat{\text{pl}}+\widehat{\text{rl}}-1)}{{(n-1)}^2}\end{array}$$(3) can be re-expressed as:

$\begin{array}{c}\widehat{\mu }=\frac{\widehat{\text{nl}}+\widehat{\text{rl}}-\widehat{p}}{1-\widehat{p}}\\ \widehat{\gamma }=\frac{\widehat{p}(\widehat{\text{nl}}+\widehat{\text{rl}}-1)+\widehat{\text{pl}}}{{(\widehat{p}-1)}^2}\\ \widehat{\alpha }=\frac{\widehat{p}(\widehat{\text{pl}}+\widehat{\text{rl}}-1)+\widehat{\text{nl}}}{{(\widehat{p}-1)}^2}\end{array}$(4)

9 These same adjusted estimates can be used when aggregating to the student instead of the question. However, the results should be interpreted carefully. As there are often fewer questions than there are students in a class, the adjusted learning types will be a noisier signal due to small numbers randomness when aggregating to the student.

10 As $n$ approaches infinity, each of the adjusted estimators becomes equivalent to their unadjusted counterparts. See equation 5.

$$\underset{n\to \infty }{\lim }\widehat{\mu }=\underset{n\to \infty }{\text{lim}}\frac{\widehat{\text{nl}}+\widehat{\text{rl}}-1}{n-1}+\widehat{\text{nl}}+\widehat{\text{rl}}=\widehat{\text{nl}}+\widehat{\text{rl}}\underset{n\to \infty }{\lim }\widehat{\gamma }=\underset{n\to \infty }{\text{lim}}\frac{n(\widehat{\text{nl}}+\widehat{\text{pl}}n+\widehat{\text{rl}}-1)}{{(n-1)}^2}=\widehat{\text{pl}}\underset{n\to \infty }{\lim }\widehat{\alpha }=\underset{n\to \infty }{\text{lim}}\frac{n(\widehat{\text{nl}}n+\widehat{\text{pl}}+\widehat{\text{rl}}-1)}{{(n-1)}^2}=\widehat{\text{nl}}$$(5)

11 Localization is the tendency for an industry’s plants to be geographically near each other. Notable examples are the software, car, and oil industry.

12 We have estimated the adjusted learning types using the macroeconomics TUCE national data with similar results (published online—https://goo.gl/BO6B4Z).