Analysis of Three Factors Possibly Influencing the Outcome of a Science Review Process

Abstract

We analyzed a process for the annual selection of a Federal agency's best peer-reviewed, scientific papers with the goal to develop a relatively simple method that would use publicly available data to assess the presence of factors, other than scientific excellence and merit, in an award-making process that is to recognize scientific excellence and merit. Our specific goals were (a) to determine if journal, disease category, or major paper topics affected the scientific-review outcome by (b) developing design and analytic approaches to detect potential bias in the scientific review process. While indeed journal, disease category, and major paper topics were unrelated to winning, our methodology was sensitive enough to detect differences between the ranks of journals for winners and non-winners.

ACKNOWLEDGMENT

The authors extend most sincere thanks and appreciation to Dr. Charles Heilig, CDC, for his sharp and deep probing, and hours of critical reviews, discussions, and analysis, all of which profoundly influenced this work.

Notes

This article not subject to US copyright law.

Color versions of one or more of the figures in the article can be found online at http://www.tandfonline.com/gacr.

1. The top-level organizational structure of the U.S. Centers for Disease Control and Prevention is composed of several national centers, one institute (i.e., National Institute for Occupational Safety and Health), and multiple offices. Hence, the acronym for this collective structure of centers, institute, and offices is CIO.

2. As an example, JIFs may not be available for both print and electronic formats, where each format is assigned a unique ISSN even for the same, common journal title; The New England Journal of Medicine is an example of this scenario, and since an electronic format JIF for The New England Journal of Medicine was not available, the print format JIF, based on the common journal title, was used in the analyses.

3. The unit of analysis is not necessarily the “element surveyed” (Nardi, 2003), which in our case was an SSA published paper in a peer-reviewed journal.

4. The reader is referred to Fig. 2, which is a graph with double logarithmic axes, as an aid to understanding the visual expression of Zipf's law.

5. Each tree carries a letter identifier. For example, the “Anatomy,” “Organism,” and “Diseases” tree identifiers are “A,” “B,” and “C,” respectively. Tuples, based on the first letter tree identifier, indicate the first level branch of each tree. “Bacterial Infections and Mycoses,” “Virus Diseases,” and “Parasitic Diseases” are identified with “C01,” “C02,” and “C03,” respectively, and are three of the first-level branches of the “Diseases” tree.