The lottery in Babylon—On the role of chance in scientific success

‘The Lottery in Babylon’ is the title of a vivid dystopian short story by Jorge Luis Borges. It envisions a society in which every individual's destiny is determined and changed every 60 days by a lottery that is ‘secret, free of charge, and open to all’ (Borges 1991 [1941], 103). The introduction of this lottery by ‘the Company’ has turned Babylonians into a speculative people wholly consumed by an ‘infinite game of chance’. It is easy to read this story as a parable to how we deal with the most existential questions: Are we here because of destiny or because of a historical accident? Am I here because I was lucky or because I deserve it? As anthropologists and sociologists have stressed time and again, every culture begins with an explanation for the contingency of its existence and, thus, with (religious) answers to these questions. This is nowhere more apparent than in the culture that we scientists have built over the last 350 years. We see the world as ruled by universal laws of nature and whenever this view is called into question, we tirelessly produce new arguments and data, to leave as little to chance as possible in our explanations. ‘God does not play dice’ was Einstein's reaction to quantum mechanics.

Science is supposed to be open to all, and, despite the infamous reviewer 2, we hold the deep-seated view that those among us who receive prestigious prizes and the most citations must deservedly be primi inter pares. As a scientist, it is difficult to not hold such a meritocratic view. Believing that success in science were only about luck or privilege, is discouraging and, thus, not just a bad motivator. It will also lead to colleagues labeling you as a fatalist and, consequentially, a morally unfit scientist. Should you believe that successful scientists have, for the most part, been lucky, you would best keep this to yourself. Explaining your own success by luck, however, is unproblematic. It will make you seem humble, or it will at least make your success more tolerable for your colleagues (Loveday 2018). In light of the steep hierarchies and the highly skewed distribution of recognition (citation numbers are as unevenly distributed as music downloads or movie revenues), a culturally supported belief in a scientific meritocracy is what keeps the peace and motivation up among scientists.

Enter: The Lottery. Funding organizations are currently introducing lotteries to science. Not ‘The Company’, but the Health Research Council of New Zealand and the Volkswagen Foundation in Germany have started using lotteries in two of their funding programs. The idea of lotteries seems seductive to many of those that are longing for change in the current power and funding structures, much like the Babylonians in Borges’ story. ‘Explorer Grants’ ¹ in New Zealand and ‘Experiment!’ ² in Germany are comparatively small funding programs, but they have already raised significant interest. The most prominent idea behind experimenting with lotteries is based on criticism on peer review. It is well established that peer review is conservative in that it disadvantages highly original ideas and puts control over science exclusively in the hands of a scientific elite. Many also see peer review as wasteful because writing, reading, and evaluating research proposals takes a lot of time. Especially funding organizations are complaining that it has become harder to find enough willing and qualified reviewers. Lotteries promise to address both issues. Lotteries are unbiased and cheap to organize. That is why other funding organizations are watching closely on these experiments. It is safe to say that large funding organizations are ready to follow suit, should these experiments be deemed successful (see, e.g. Egger 2018). It is easy to see why: it is much easier to organize a lottery than finding highly qualified reviewers, get them to read the proposals, get them to write substantial reviewer reports, maybe even fly them to a panel meeting, and then deal with disgruntled applicants, complaining about badly justified rejections. What often gets lost here, is that being excited about lotteries may come at the cost of efforts to make peer review better, not least because the latter is harder to do than the former.

While funding organizations seem to have a clear (and mainly economic) interest in introducing lotteries, scientists themselves may be more skeptical; see the comments to a recent news story in Nature unmasking lotteries as ‘a bureaucratic short-cut’ (Beattie 2020) and a way to ‘stall ideas and careers’ (Vindin 2020). The universe can be non-deterministic but science must be meritocratic, remember? In fact, scientists may be far more critical than funders and the news media anticipate. A preliminary analysis of a 2020 representative survey of German scientists shows only 15% being strongly or moderately in favor of lotteries, while 70% of the 3700 respondents were either strongly or moderately against it. The opposition is similar across all disciplines, with younger scientists being more critical than the rest. ³ The general skepticism among researchers towards lotteries stems, at least in part, from the worry that careers will become even more speculative, in a situation where they are already seen as too speculative. So far, proponents of lotteries have done little to address these worries.

Whether scientists can be convinced that lotteries ‘actually work’ remains to be seen. The Volkswagen Foundation is funding a research project to accompany their experiment, which combines a lottery with the old review process as a control. ⁴ There are also a handful of speculative papers out making the case for lotteries in science funding (Fang and Casadevall 2016; Avin 2015, 2018; Roumbanis 2019). It may thus seem that the introduction of lotteries in science is based on good reasons and accompanied by a scientific effort to produce evidence for (or against) its effects. We think, however, that the promise of reducing the burden on funding organizations and reducing bias in peer review are very weak reasons for what is at stake here.

So, what is at stake here? It is true that funding organizations have drawn justified criticism, that their application and reviewing procedures have become too burdensome (looking at you, ERC). It is also true that scientists would prefer if as little money is spent on organizing review procedures, because this is money not spent on actual research. Keeping the administrative parts of review procedures as lean as possible is non-controversial and if lotteries reduce administrative burden, that is all for the better. (It has to be shown in practice, of course, that lotteries are indeed less costly administratively.) However, review procedures are not just a burden that should be minimized. They also have real scientific value. Writing applications, reading them evaluatively, deliberating on their merit and providing feedback to applicants are integral parts of the scientific process. These are as important as gathering data, performing experiments, or writing papers. They also expand, contextualize and structure knowledge. Not least, they provide the organized skepticism that many see vanishing in times of increased competition and a publish-or-perish research culture. Engaging critically and constructively with the work of colleagues benefits applicants and reviewers equally and keeps up standards of quality. It may seem that lotteries will distribute money and status more efficiently, but the cost of reducing organized skepticism will have to be factored in.

It is not just organized skepticism that has to be factored in, however. It is not enough for funding decision to be right, they also have to be legitimate; and on legitimacy peer review clearly outperforms lotteries. Decisions on scarce resources (e.g. funding or publication space) are seen as legitimate when they are the result of critical deliberation according to scientific criteria. It is this process of invoking scientific judgment and deliberating on open epistemical issues, which is decisive in distinguishing between scientifically generated knowledge and public opinion or mere assertions. The authority and interpretative power of scientific knowledge rests on this, as reasonable decision making is a crucial contribution of science to modern societies. At least since the Enlightenment, this notion of a legitimate decision refers to the (implicit) conviction that intelligent beings are competent to judge to the best of their knowledge and belief. Giving up such ideals, just because they cannot be fully realized, seems short-sighted. Giving up on expert judgment just because it is hard to do well, is not only an unnecessary retreat but also a fundamental reversal in normative orientation. Younger scientists, who are fatalistic about their careers, will be among the first to notice such a reversal, along with the climate change deniers, anti-vaxxers and flat-earthers. The former will exit quietly, while the latter will be louder and politically more effective, one should fear.

These are issues of legitimacy that scientists either tend to confuse with or hold as neatly separable from issues of objectivity. By necessity, funding decisions lack objective validity as scientific research operates at the margin of knowledge and ignorance; even more so for research that is only planned and not yet performed. It is a distorted argument to deduce from this, that peer review is insufficient to ground decisions. It would stipulate objectivity as a necessary criterion for scientific judgment. This mostly implicit premise reveals a rather dogmatic epistemology with two extreme alternatives: either, an evaluation relies on fully objective judgments, or, if there is only impartial validity (i.e. bias), a lottery is the better alternative. These dichotomist considerations are stuck in a false dilemma. It results from an illusion of justice, conceived to be guaranteed by rigorously formalized and automated procedures; i.e. either by drawing lots as the perfect mechanism of impartiality or, less frequently suggested, by trained artificial intelligence (Brezis and Birukou 2020). However, this formalizes justice to an extent that leaves equal distribution as the only just result. As such, procedural justice and deliberative scientific judgment are stripped of their potential to create and ground values. The form of government resulting from such an ideal is one of (pure) chance provided through a (perfect) algorithm. It treats all who are subjected to it equally, but equally unfairly. Such a policy is, ultimately, nothing less than a capitulation to the challenges of well-founded governance based on justifiable and comprehensible decisions and, instead, the acceptance of a regime of indifference.

Arguing that lotteries are more efficient and could remedy the conservatism of peer review also risks overlooking the fact that a new system of control would be put in place. Efficiency will mean that a smaller number of people will be involved in reaching funding decision, with a high chance that most of them will not be scientists or experts but administrators or politicians. They would organize the lottery, allocate the lots fairly, transparently and equally accessible to all participants, in order to ensure equal opportunities. Depending on who organizes the lottery, decides on who gets a lot, how many lots are awarded in total, how equal access is granted for all, it takes little imagination to see that the influence of specific interests cannot be ruled out here either. Biases and partiality will occur here as well. As a result, the question is not how we eliminate bias, but whether we prefer the biases of scientists over the biases of politicians and administrators.

Finally we should ask, why are lotteries discussed as possible alternatives now? Shortcomings of peer review, such as conservatism, bias and inefficiency have always been there and well known. More ironically, we have known for at least 40 years that peer review decisions already are in large part a lottery (Cole, Cole, and Simon 1981); why, then, do we want even more of it? What proponents of lotteries are, thus, saying is not ‘Let's add a little randomness to balance out existing biases’, but, ‘Let's add more randomness to solve the problems of the already existing randomness’. Should we not ask instead why funding agencies—more than individual scientists—want to delegate scientific judgment to randomizing algorithms? Why are the same actors, that recommend better incentives for doing robust scientific work, also willing to incentivize pure speculation with lotteries?

In our view, answering these questions will require a more thorough understanding of scientific cultures than a merely economic one, reflected in the talk of incentives, efficiency, bias, and hollowed-out concepts of justice and legitimacy. The long standing and varied debate about science and democracy would be the right context to ask how we change the (self-)governance of science without unnecessary harm to scientific cultures. One could start by relating the emerging platform cultures in science to the societal trend of delegating power to algorithms and digital platforms (Mirowski 2018). Or one could start by empirically observing what actually motivates scientists or makes up scientific cultures and, maybe, one would find that increasing competition and fatalism strangely coincide with the lottery fad (Loveday 2018). Or one could balance the pervasive deficit view of peer review with attempts to also show the positive small and large scale effects of organized, professional criticism (Lamont 2009). Or, finally, one could go back to Borges’ story and wonder whether ‘Babylon is nothing but an infinite game of chance’. ⁵

Acknowledgement

We acknowledge support by the Open Access Publication Fund of Humboldt-Universität zu Berlin.

Notes

1 http://www.hrc.govt.nz/funding-opportunities/researcher-initiated-proposals/explorer-grants

2 https://www.volkswagenstiftung.de/en/funding/our-funding-portfolio-at-a-glance/experiment

3 See DZHW Scientist Survey: https://www.wb.dzhw.eu/en/index_html

4 Their PR efforts also extend to twitter with #PeerReviewLottery

5 Or, really finally, we may throw that book in the fire, as Homer Simpson does with Shirley Jackson's ‘The Lottery’, an echo to Borges' story, in the episode ‘Dog of Death’ from The Simpsons.