Evaluating Journal Quality: Beyond “Expert” Journal Assessments in the IS Discipline

Abstract

Assessing the relative stature of journals devoted to the information systems (IS) discipline is an important issue for IS scholars and those who evaluate them. Even though journal assessment results are often dubiously applied by those making hiring, promotion, and merit decisions, the fact that they are so often a major ingredient in these decisions demands that we understand underlying journal assessment processes. Beyond processes involving the opinions of various “experts,” we here examine how IS journals can be evaluated based on overt behaviors of crowds of IS scholars. These behaviors are revealed preferences, in contrast to stated preferences found in opinions. Two classes of objective journal assessments are studied: impact measures and power measures. Among the former, we find that so-called journal impact factors are problematic, rendering their meaningfulness in evaluating journal stature highly suspect. Another kind of impact measure, the H-index, is found to be a more straightforward way to gauge journal impact. Two power measures for assessing IS journal stature are examined: publishing intensity and publishing breadth. The stature of IS journals according to each of the impact measures and power measures is determined. A comparison of the results shows that a small group of four or five IS journals are repeatedly found at the top across multiple objective assessment approaches. To account for both the consumption and production of IS research, it is suggested that a combined use of impact and power measures be employed in exercises aimed at evaluating relative statures of journals devoted to IS research.

Keywords:

• bibliometrics
• breadth
• citation
• H-index
• impact factor
• impact measures
• intensity
• journal evaluation
• journal quality
• journal stature
• power measures
• revealed preference

Notes

¹The private research-intensive universities are Boston University, Brandeis, Brown, Carnegie Mellon, Case Western Reserve, Chicago, Columbia, Cornell, Dartmouth, Duke, Emory, George Washington, Georgetown, Harvard, Johns Hopkins, Massachusetts Institute of Technology, Miami, New York, Northwestern, Notre Dame, Pennsylvania, Rice, Southern California, Stanford, Syracuse, Tulane, Vanderbilt, Wake Forest, Washington (St. Louis), Yale, and Yeshiva.

²The public research-intensive universities are Arizona, California-Berkeley, California-Davis, California-Irvine, UCLA, California-San Diego, Cincinnati, Colorado-Boulder, Florida, Georgia, Georgia Tech, Illinois, Iowa, Maryland, Michigan, Michigan State, Minnesota, North Carolina, North Carolina State, Ohio State, Penn State, Pittsburgh, Purdue, Rutgers-New Brunswick, University at Buffalo, Texas-Austin, Texas A&M, Utah, Virginia, Washington-Seattle, and Wisconsin-Madison.