Mining Process Data to Detect Aberrant Test Takers

ABSTRACT

Item harvesters who memorize, record and share test items can jeopardize the validity and fairness of credentialing tests. Item harvesting behaviors are difficult to detect by the existing statistical modeling approaches due to the absence of operational definitions and the idiosyncratic nature of human behaviors. Motivated to detect the hard-to-define aberrant test-taking behaviors like item harvesting, we proposed a data-mining approach that utilized the process data and identified the examinees whose test-taking processes deviate from the majority of examinees. Specifically, two steps were implemented in the proposed approach: First, archetypes of test-taking processes are learned with the k-means clustering algorithm; second, examinees whose behavioral patterns deviate from the archetypes are flagged for further investigation. Given that the process data makes it possible to capture more subtle differences between the aberrant test-takers and normal examinees, the proposed approach is expected to be complementary of the statistical modeling methods, capture additional types of aberrant test-takers and increase the probability of discovering the elusive item harvesters.

KEYWORDS:

• Process data
• cheating detection
• abnormal test-taking behavior
• clustering
• data mining

Acknowledgements

We would like to thank Financial Industry Regulatory Authority (FINRA) for supporting this research, and the editor and reviewers of Measurement: Interdisciplinary Research and Perspectives for suggestions that improved this paper.

Notes

1. The means and standard deviations of response time are in terms of log-transformed seconds. A log transformation was employed to eliminate the strong positive skew of the raw RTs.

2. The qualitative results learned from the “initial item response” stage are presented in Figure A1 in Appendix. Cautions need to be taken when interpreting and using the results from the “initial item response” stage.

3. The label switching issue occurs when the label of the same cluster centroid changes across different subsamples. To resolve the issue, K’ clusters in each subsample (except the first subsample) are permutated such that the sum of the diagonal elements in the two-way frequency table (cross-tabulation of this subsample with the first subsample) is maximized.