A theoretical basis for accepting undergraduate academic record as a predictor of success in a research career. Implications for the validity of peer review

A major goal of a research granting agency is to select those researchers who will best advance its mission. Noting a poor correlation between undergraduate academic record and subsequent success in medical research, agency officials have argued that less reliance should be placed on academic record. However, from the same data, a contrary conclusion can be drawn. Consider the existence of two evaluation systems, A (teacher evaluation), and B (peer review). It is assumed here that at the time of undergraduate education, A applied to a group of individuals on repeated occasions would generate a consistent rank order. Similarly, B applied to the same group of individuals would generate a consistent rank order, which would usually be different from the A‐generated rank order. At a later point in time, A is not feasible. Grants are then awarded based on B. Thus, individuals who score highly under B prosper. I propose that, since rank orders obtained on different occasions using different evaluation systems may not be well correlated, it is not surprising that there is a poor correlation between early A success and later B success. However, this disregards the real issue. Which evaluation system is the best predictor of the ability to advance agency goals? Because A is not feasible at the later time‐point, it does not follow that we should discard the results obtained earlier. Rather than questioning the reliability of A applied earlier, agency officials should be questioning the reliability of B applied later.

Keywords:

• grant agency
• evaluation systems
• rank order variation