Measuring the Reputation and Productivity of Communication Programs

Abstract

Recent years have seen increasing interest in assessing the prestige and productivity of programs in communication. An NCA doctoral reputational study was published in 2004, and a new National Research Council assessment is underway that will for the first time include the communication field. Two systems relying on objective appraisal of faculty productivity have been launched since 2005, one by the Academic Analytics Corporation and the other by the Communication Institute for Online Scholarship (the ComVista database). This study details similarities and differences among the four systems’ methodologies raising questions about features of these studies’ strategies for sampling and measurement. ComVista ranks, based on objective assessment of publication productivity, were highly correlated with the subjectively based NCA reputational ranks. The common practice of restricting assessment of publication productivity to a small span of years reduced this correspondence. Recommendations are provided for improving future assessments.

Keywords:

• Reputational Studies
• Publication Productivity
• ComVista
• NRC
• FSPI
• NCA

Notes

1. Prior to the 1990s, Renee Edwards and associates conducted a series of reputational studies of doctoral programs in the speech communication field (e.g., Edwards & Barker, 1977, 1979a, 1979b, 1983, 1984; Edwards & Pood, 1987; Edwards, Watson, & Barker, 1988, 1989; Watson, Edwards, & Barker, 1989).

2. The NRC methodology study (Ostriker, Kuh, & Voytuk, 2003) acknowledges that a program's reputation may change more slowly than the indicators on which it is founded and recommends annual recalculation and republication of some of the NRC data.

3. The NRC methodology will correct for faculty size (L. Putnam, personal communication, December 6, 2007). The FSPI corrects for comparisons among disproportionately sized units in two ways: first, by limiting its sample to units that are at least 50% of the median size of all the programs sampled (thus eliminating truly small units), and second, by reporting productivity in rates of output per faculty member.

4. The NRC Methodological Study reports dissatisfaction with the latter two questions and recommended they be dropped from the 2007/2008 NRC assessment. The question on educational effectiveness was deemed redundant of the question on research quality and the question on change in program was deemed better assessed by comparisons between actual NRC analyses (e.g., between the 1995 NRC study and the current study).

5. The NRC Methodological Study acknowledges there may not be adequate data sources for all fields. Awards are only assessed for faculty in the arts and humanities. Publication and citation data are only assessed for faculty in the sciences.

6. The FSPI counts books as five times the weight of a journal article but permits subscribers to its rankings to customize which intellectual products are included and what weights they receive.

7. But this does not solve the problem with regard to citation counts.

Additional information

Notes on contributors

Timothy D. Stephen

Timothy D. Stephen (Ph.D., Bowling Green State University, 1980) is a professor in the Department of Communication at University at Albany in Albany, New York