Utilizing Item Analysis to Improve the Evaluation of Student Performance

Abstract

One of the cornerstones of teaching is the evaluation of student performance. Traditionally, such evaluations are performed through the administration of exams, quizzes, research papers, and group projects. Although faculty are accustomed to evaluating students, rarely do they evaluate the quality of the aforementioned methods used to assess student knowledge. The present study will illustrate how certain measurement theory techniques (i.e., item difficulty, index of discrimination, Cronbach's alpha, and point biserial correlation) can be utilized to investigate the reliability and validity of student performance and what their impact is on grade distribution. Additionally, a new method for performing item analysis is proposed. To demonstrate the applicability of measurement theory in estimating the reliability and validity of student performance, we draw from one of the author's experiences in teaching an introductory course in political behavior.

Keywords:

• item analysis
• student assessment
• student performance

Notes

The databases surveyed are ArticleFirst, ERIC, GenSciAbs, JSTOR, PapersFirst, SocialSciAbs, and WorldCat. The keywords used for this search were “political science” in combination with “classical test theory,” “item response theory,” and “test development.”

Although Item Response Theory (IRT) was initially considered for inclusion in this article, the minimum sample size requirement (i.e., 200 examinees according to Crocker and Algina [1986] and several hundred according to de Ayala [2009]) makes the use of IRT for this study unfeasible.

The terms “items” and “questions” are used interchangeably throughout this article.

Cronbach's alpha and coefficient alpha are used interchangeably throughout the article.

Since the statistical distribution of a dichotomous item (e.g., correct/incorrect) conforms to a Bernoulli distribution whose variance is given by the probability p of attaining the desired outcome (correctly answering the item) times one minus the probability, p(1−p), it can be proven that setting the item difficulty p to .5 maximizes the variance.

Given that the test is comprised of 50 items, a standard of 60% connotes that a student will fail the test if they correctly answer less than 30 items on the test. Furthermore, since each item corresponds to a Bernoulli process (i.e., a dichotomous variable) then the scores on the test must conform to a Binomial distribution. Then, given that the probability of a correct response is p = .625, one can compute the probability of obtaining a failing grade using the Binomial cumulative distribution function, where n denotes the number of items on the test, k denotes the number of items answered correctly, p denotes the item difficulty (probability of correctly answering a question), and t denotes the upper limit of the sum. Plugging the numbers of the problem into the equation yields F(29) = 0.302. That is, on average, 30.2% of students will fail the test if all the item difficulties are equal to 62.5%.

The following 100-point grading scale was employed: the range [93,100] denoted the grade A, [90,93) denoted an A−, [87,90) denoted a B+, [83,87) denoted a B, [80,83) denoted a B−, [77,80) denoted a C+, [73,77) denoted a C, [70,73) denoted a C−, [67,70) denoted a D+, [63,67) denoted a D, [60,63) denoted a D−, and [0,60) denoted an E (failure). Square brackets denote the range included the number while parentheses denote the range excluded the number.

The correct answer is marked with an asterisk.

The final course grade included scores from quizzes, class participation, and group projects. However, for illustrative purposes, it is simpler to imagine that the final grade is comprised only of the three exam scores.