Dynamic Bayesian Network Modeling of Game-Based Diagnostic Assessments

Abstract

Digital games offer an appealing environment for assessing student proficiencies, including skills and misconceptions in a diagnostic setting. This paper proposes a dynamic Bayesian network modeling approach for observations of student performance from an educational video game. Drawing from and advancing methods in dynamic Bayesian networks, cognitive diagnostic modeling, and analysis of process data, a Bayesian approach to model construction, calibration, and use in facilitating inferences about students on the fly is described, and implemented in the context of an educational video game.

Keywords:

• Dynamic Bayesian network
• game-based assessment
• diagnostic assessment

Notes

1 There are a few other possible behaviors that were identified by the cluster analyses (see Kerr et al., 2011; Kerr & Chung, 2012a) having to do with the game mechanics (e.g., getting a key to open a lock). Attempts with these results were ignored from the current analysis because they were associated with game mechanics rather than underlying mathematics proficiency, such that there are not firm beliefs about the evidentiary relevance of these behaviors for inferences about students’ mathematics proficiencies.

2 This constraint may be relaxed if desired (Reye, 2004). Note that mastery here takes on the meaning of success with levels that measure the skill in the game. This is not to say that if the student is a master of a skill in the game, then they are master outside of the game, as the latent variables are interpreted in the context of the game and its modality (Mislevy et al., 2014). Of course, it may be desirable for the interpretation of performance in the game to generalize to other contexts. The use of a rigorous design process lends support to this, but further empirical work relating the characterization of student performance in the game to performance outside of the game is beneficial. Such relationships between in-game performance and external measures are discussed in a later section.

3 Specifically, attempts were made to calibrate all the levels in a single analysis using JAGS software (Plummer, 2003) on a computing cluster dedicated to running long jobs, with 12 cores with 4000 MB memory per core. However, the computing capabilities were exhausted before the necessary number of iterations could be run.

4 A third phase is possible, in which the model-based expectations for performance at the future time points are obtained. This is given by the posterior predictive distribution; for example the next time point X_t+1, $p(X_{t+1}|X_t)={\sum }_{{\mathbf{\theta }}_{t+1}}p(X_{t+1}|{\mathbf{\theta }}_{t+1})p({\mathbf{\theta }}_{t+1}|X_t).$ This distribution represents the updated beliefs about the student’s performance on the next attempt, given the just updated beliefs about student proficiency for the next attempt. A fourth phase is also possible, in which the posterior probability for proficiency at earlier time points is obtained (e.g., the posterior probability for the latent variables at time t − 1 given the new observation at time t). These posterior retrodictive probabilities tend not to vary much from those obtained in the first phase in (17). For this reason, and the desire to avoid the heavier computational burden associated with storing and updating all these distributions, these are not employed in the current work.

5 Though they would need to be if posterior retrodictive probabilities for the latent variables were of interest.

6 Numerical instability in the calculations of the logit meant that the correlation with the posterior probabilities for Avoiding Math was calculated on the untransformed probabilities.