Meritocracy Voting: Measuring the Unmeasurable

Abstract

Learned societies commonly carry out selection processes to add new fellows to an existing fellowship. Criteria vary across societies but are typically based on subjective judgments concerning the merit of individuals who are nominated for fellowships. These subjective assessments may be made by existing fellows as they vote in elections to determine the new fellows or they may be decided by a selection committee of fellows and officers of the society who determine merit after reviewing nominations and written assessments. Human judgment inevitably plays a central role in these determinations and, notwithstanding its limitations, is usually regarded as being a necessary ingredient in making an overall assessment of qualifications for fellowship. The present article suggests a mechanism by which these merit assessments may be complemented with a quantitative rule that incorporates both subjective and objective elements. The goal of “measuring merit” may be elusive, but quantitative assessment rules can help to widen the effective electorate (for instance, by including the decisions of editors, the judgments of independent referees, and received opinion about research) and mitigate distortions that can arise from cluster effects, invisible college coalition voting, and inner sanctum bias. The rule considered here is designed to assist the selection process by explicitly taking into account subjective assessments of individual candidates for election as well as direct quantitative measures of quality obtained from bibliometric data. Audit methods are suggested to mitigate possible gaming effects by electors in the peer review process. The methodology has application to a wide arena of quality assessment and professional ranking exercises. Some specific issues of implementation are discussed in the context of the Econometric Society fellowship elections.

Keywords:

• Auditing peer review
• Bibliometric data
• Election
• Fellowship
• Measurement
• Meritocracy
• Peer review
• Quantification
• Subjective assessment
• Voting

JEL Classification:

• A14
• Z13
• C18

ACKNOWLEDGMENT

My thanks to four referees, the Editor Essie Maasoumi, Steven Durlauf, John Rust, and Peter Schmidt for comments on earlier drafts and some useful additional references. Olav Bjerkholt kindly guided me to original source material related to Ragnar Frisch and the Econometric Society.

Notes

¹The original source was Frisch's examination article in public finance in 1919, a document that has survived but has never been published. Olav Bjerkholt has provided the following literal translation of the original:

“Man must not be deterred by the apparently impossible. History has shown that the human beings have had a wonderful ability for obeying the maxim of Aristotle: ‘Make the unmeasurable measurable.’”

These words appear to be the first words written by Frisch to be published. Remarkably for an examination article they formulate, as Andvig and Thonstad (1998, p. 6) put it, “his own overriding future research policy” to tackle the apparent impossible. Much of the early empirical econometric work on measuring demand elasticities (e.g., Schultz, 1924) may well have appeared in the same light at that time.

²The 1950s debate was prompted by correspondence of Oscar Morgenstern circulated in 1953 to all fellows of the Econometric Society stating that

in my view the Fellows ought to be persons who have done some econometric work in the strictest sense. That is to say, they must have been in one way or another in actual contact with data they have explored and exploited, for which purpose they may have even developed new methods.

This viewpoint was strongly supported by some fellows (among them Robert Geary, Charles Roos, and P. C. Mahalanobis) and opposed by others (including Tjalling Koopmans and Jacob Marschak). In the end, no changes to criteria or procedures for fellowships were made.

³In a letter to the President of the Econometric Society on June 26, 2010, the author and David Hendry raised concerns about the role of econometrics within the ES, pointing to

a mounting concern that the Econometric Society has become progressively less representative of econometricians within the society with consequential impacts, particularly on the careers of younger econometricians. To many there is an emergent crisis in econometrics because of the lack of acknowledgement and representation and the growing difficulties econometricians have in publishing in Econometrica and other general interest journals in economics. The movement away from econometrics is manifested each year in the election of officers, council and fellows, the appointment of Editors of Econometrica, and recently by the formation and nature of the new journals. While the concern over under-representation has occasionally been raised in Fellows Meetings at various Econometric Society conferences since the 1980s, the situation seems to many to have grown considerably worse over the past decade. Many people are now puzzled about the role of the Econometric Society in terms of what it does for econometrics and the increasing lack of congruence between its name and its focus. By contrast and in response to the direct needs of the econometrics community, a large number of highly successful regional and thematic meetings have been organized that are outside the aegis of the Econometric Society and continue to grow and prosper without any connection to or support from the Econometric Society.

⁴From archival research on correspondence among the Council of the Econometric Society in the early 1930s, Louça (2007) reports that one candidate for a fellowship was opposed on the grounds that “he would not know a partial derivative” (p. 31), an injustice as it turned out. Bjerkholt (1998, p. 53 and footnote 32) provides original source material and further details on this particular incident. Another candidate was repeatedly opposed as President of the Society as “not recommendable” on the grounds that he “uses many words to express his meanings” (Louça, 2007, p. 35).

⁵Some journals in economics already use automated measures in determining fellowships (Journal of Econometrics), distinguished authorships (Journal of Applied Econometrics), and annual prizes (Econometric Theory). The measures employed in these awards rely on bibliometric counts and are not complemented with peer review data.

⁶Even when bibliometric data are not explicitly used in the preparation of research assessment exercises or other official rankings, that data may already appear indirectly through individual use in providing “peer esteem” evaluations that many ranking reports request. The use of automated software that universities now provide for harvesting publication information also implies that bibliometric data are being routinely utilized in report preparation.

⁷See Chang et al. (2012) for a recent overview of various bibliometric research assessment measures and an empirical application of the methodology to journals in econometrics and statistics using journal citation reports and the Thomson Reuters ISI Web of Science database.

⁸To be measured a quantity must first be defined, a task of longstanding relevance in economics whether the quantity in question is the price level, utility, happiness, or merit. In all such cases the inherent multidimensional nature of the quantity (and in the present case the subjective elements involved in defining merit) must be addressed even when a single index proxy variable is used in the accounting process. More subtle is the fact that, however difficult the accounting definition may be, it is still far easier than defining an appropriate probability space for “measuring” the said quantity and using it in statistical work. Almost always in economics and other social sciences, the probability space is too limited for there to exist any “true” representation of the measure (or true data generating process (dgp)), a problem that is still largely untouched in econometrics (see Phillips, 2005, for further discussion).

⁹A request for data on the annual Econometric Society fellowship election outcomes over 1990–2011, including the votes received by each of the nominated candidates, was denied. These data would enable empirical research on ES data of voting formulae such as those considered in this article, the effects of various parameterizations, and some counterfactual analysis.

¹⁰The process is analogous to the peer review process in academic journals, where referee reports and recommendations are evaluated by an associate editor or co-editor and editor before an editorial decision is made. Referees often voluntarily return in their cover letters an indication of their own expertise in the field, rather like R _jia .

¹¹In working out the exact distribution theory, it is convenient to let the aggregate publication component Y = 2n ₁ + n ₂ have a continuous density. A corresponding discrete equivalent can be computed by integration over cells of unit length covering the integers.

¹²Peer evaluation, as embodied in X _b , is surely influenced by accomplishment, as measured in X _a . So we can expect X _a and X _b to be dependent in general. In a large elector population, however, there may be many electors who have little knowledge of the accomplishment of candidates outside their fields, in which case the assumption of independence may not be fatally violated. See the discussion in Section 7 on the Econometric Society fellowship elections, where the Council of the Society noted that “works of several well known nominees had been read by only a few Fellows” and “ballots show that some nominees failed of election primarily because their work was to a large extent unknown to the Fellows.”

¹⁴A single sentence may be sufficient, such as “I know the candidate's work well, can confirm that the papers listed in the nominating statement are significant contributions, and am strongly supportive of election,” or “I have read several of the candidate's papers, find the research to be incremental and believe it is too early to support election of this candidate.”

¹³As indicated, items b, c, and d are voluntary. Where the information is not provided a default assessment is created as discussed in Section 3.

¹⁵Wikipedia reports that the U.S. Library of Congress adds 5 terabytes (TB) of information per month, that internet traffic is estimated to be around 160 TB a second, that Germany's climate research centre generates 10,000 TB of data per year, that Wikipedia itself had a 5.87 TB dump of raw data in January 2010, and that Google processes about 24 petabytes of user-generated data per day.

¹⁶Proposals to expand the list of ES journals go back a long way. For example, to reflect the substantial expansion and growth in diversity in the discipline by the 1980s, the author proposed to the society in 1982 that Econometrica be expanded into two flagship journals, one subtitled in Economic Theory, and the other in Econometrics, much as the Institute of Mathematical Statistics created the Annals of Probability and Annals of Statistics out of the original Annals of Mathematical Statistics.

¹⁷At present, there is no requirement in the online system for electors to look at the nominating statement before voting.

¹⁸For further reading on Frisch, see Bjerkholt (1995, 2005, 2015).

¹⁹Correspondence between Irving Fisher (President of the ES) and Ragnar Frisch (Editor of Econometrica) provided guideline criteria for the selection of Fellows. In a letter to Council on 12/01/1932, Fisher appended a statement listing 5 qualification categories for selecting fellows: “1. They should be economists; 2. They should be mathematical; 3. They should be statisticians; 4. They should have made some original contributions; 5. Some of these contributions should be in economic theory.” In a response to Fisher on 1/11/1933, Frisch indicated agreement on these criteria, which he re-articulated as: “1. The candidate must be an economist acquainted with economic theory; 2. He must have a mathematical foundation; 3. He must have some knowledge of statistics; 4. He must have done some original work; 5. Some of this original work must have been in economic theory.” This correspondence is contained in the Frisch correspondence collection of the National Library of Norway. I am obliged to Olav Bjerkholt for sending me copies of this correspondence.

²⁰These qualifications for fellowship of the ES are very close to the requirements for membership of the society that were originally laid out in a letter of June 17, 1930 to multiple recipients from Irving Fisher, Charles Roos, and Ragnar Frisch (from the Ragnar Frisch Archive, the National Library of Norway). These requirements limited membership to those who: “(a) are thoroughly familiar with general economic theory; (b) have a working knowledge of mathematics as applied to economic theory and statistics; (c) have some knowledge of accounting; and (d) have published an original contribution to economic theory or to the analysis of such economic statistics or accounting as have a definite bearing on problems in economic theory.” The reader is referred to Bjerkholt and Qin (2010) for further discussion of this history, ES membership elegibility, and related matters.

²¹As one referee put it, “my impression is that a significant number of fellows do not vote in an election.”

²²The interpretation of citation data has spawned a large literature that reveals the sociological complexity of the practice. For example, in her study of the factors involved in citation, Cozzens (1989) notes that the recognition/reward element of citations neglects rhetorical factors such as convenience (“the easier it is to fit the knowledge claims of the article into the rhetoric of later papers, the more likely the article is to be cited”); and Brooks (1986) reports empirical evidence from author surveys covering a wide range of academic departments that 70% of the citations surveyed arose from a “complex interplay of multiple citer motives” involving positive credit, negative credit, and service to the reader elements, some of which may be conflicting.

²³Econometric Society website is http://www.econometricsociety.org/society.asp.

²⁴American Economic Association website is http://www.vanderbilt.edu/AEA/disting_fellows.htm.

²⁵European Economic Association website is http://www.eeassoc.org/index.php?page=21.

²⁶IMS official website is http://www.imstat.org/awards/fellows.htm.

²⁷Royal Statistical Society website is http://membership.rss.org.uk/main.asp?page=1280.

²⁸American Statistical Association website is http://www.amstat.org/.

²⁹See http://www.amstat.org/fellows/nominations/pdfs/RatingofNominees.pdf.

³⁰American Statistical Association website is http://www.amstat.org/careers/fellows.cfm.

³¹The Royal Society website is http://royalsociety.org/about-us/fellowship/election/.

³²British Academy website is http://www.britac.ac.uk/fellowship/elections/elecproc.cfm.

³³National Academy of Sciences website is http://www.nasonline.org/.

Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/lecr.