Leigh deserves accolades for bringing into focus the importance of how data are collected, hopefully convincing others that work on the front end, before data collection, will pay off. He successfully argues for and motivates the use of randomization through a rich collection of stories of successes and failures—and this fact alone makes me want to recommend the book to others. Analysis on already-collected data, often with no mention of their origin, receives far too much attention relative to the importance of design. Any writer moving the spotlight to pre-data collection planning deserves applause.
Randomistas successfully hits home with a general audience, which is expected given the author’s unique path as an economics professor turned Australian politician and author of other broad audience books. I left the book lying around and two people with no direct connection to research or study design picked it up and said they enjoyed their taste of it, finding it interesting and easy to digest. It is also likely to be effective for motivating a CEO-like audience—those calling the shots about whether randomization will be employed. It is important to remember, and actually hard to forget while reading the book, that it is written by an economist, not a statistician. Stories are selected and told through an economic lens—with a focus on use of random assignment in the design of natural field experiments to ultimately improve profits, save money, or justify programs. The term “radical researchers” is in the title, but radical businesses and non-profits is probably a more apt description.
I am a firm believer in the inferential power of randomization and also a firm believer that design principles, in general, are often neglected. Design sits quietly in the shadows of the coolest new methods, algorithms, and computational tools for existing data. Design is in need of well-spoken and well-known advocates who speak the language of those with money and power. So … why do I struggle with getting fully on board with recommending this book, at least without some real disclaimers? I think the most blatant reason is that the stories are wrapped up as if simply using randomization magically rids the situation of any uncertainty, providing a crystal ball view to the truth. Design is incredibly important, but randomization alone does not buy proofs and truths. More generally, wording used throughout the book perpetuates common mischaracterizations and misconceptions of what statistical inference is reasonably capable of.
However, I had to wonder—given the lofty goal of the book—is some level of over-stating conclusions actually worth getting more people to buy into the importance of careful design and randomization? Maybe. But, how much exaggeration is okay in a do-no-harm sort of way? At what point do the negative consequences outweigh the positive effects of a little simplification and over-selling? I found myself stuck in this quandary as I made my way through the book, on an emotional roller coaster between highs of gratitude for what was being said to lows of real frustration. Was I being too harsh? Too picky? I read with a pencil in hand, underlining each place where I felt uncomfortable with wording, usually because study conclusions were over-stated, misrepresented, or under-justified relative to the information provided. Uncertainty inherent in the process of getting to the conclusions was just missing in action. My copy of the book has graphite marks on about a quarter of its pages (despite my daughter’s admonitions about writing in books)!
To successfully hit what I would call a nonresearcher audience, I fully understand some simplifying and selling is needed. There are many great quotes and wise words that I agree with. However, I write this review for a Statistics journal from a statistician’s point of view, and one who is quite concerned about misuses of statistical methods and mischaracterizations of statistical inference. And, those are often related to oversimplifying the process of statistical inference and overstating conclusions—as demonstrated in this book. I talked to another statistician who had read part of the book and she expressed the same mixed-emotion reaction—so it’s not just me!
A great test of a book about a statistical topic is asking myself whether I would recommend it to students and add it to a suggested readings list. And if so, would discussion be necessary to adjust the ultimate take-home message? In this case, some of the stories could be great examples and motivators for discussion of design in a classroom, particularly for students interested in social science and economics. If used as a motivator for the importance of design, as well as lessons for how we tend to over-simplify and under-justify conclusions based on data, it could be a good one. The stories as told in the book would need that added balance—with fairly high-level discussions on a long list of topics: the difference between statistically significant and practically meaningful; applying arbitrary thresholds for declaring results as “significant”; our tendency to create and use false dichotomies in analysis and conclusions; describing results based on averages as if they apply to every and any individual in some large population (like the “average American”); more background for understanding counterfactuals relative to what we actually get from experiments; other important design concepts such as blocking and identifying the experimental unit; strategies for not stating estimates as facts; an accurate portrayal of the evolution of the history of the use of statistical methods in science; what constitutes proof in science and how it relates to inferences based on probabilistic methods; and the crucial dependence of inferences on chosen measures and instruments with inherent limitations.
I apologize for the lengthy list, but the last item is worth a little more time. Choices of what and how to measure are an integral part of the process and results can clearly be sensitive to these—particularly in social science applications where the quantity of interest is impossible to measure directly and researchers must rely on instruments with many limitations. The firmness with which conclusions are stated in this book, seemingly justified merely by randomized experiments, does not acknowledge this huge source of uncertainty and its potential impact. Such choices are often subject to the values and perspectives of people with power and the money to design the study, as well as their imperfect instruments. In Randomistas, the particular outcome used may be missing all together from statements about whether the program had an impact (or not)—operating in the unrealistic, though tidy, world of black and white. The cases are presented as open, and then closed, based on the experiment described.
In Chapter 11, titled “Building a Better Feedback Loop,” Leigh briefly wades into important topics such as ethical concerns associated with randomization, the replication crisis and its connection to thresholds of statistical significance, and preregistration of studies. I was relieved to see ink given to these topics, though their treatment is not intended as stand-alone introductions to the issues. The book ends with “Ten Commandments for Running Your Own Randomised Trial.” To be honest, I was hoping to see “find a statistician to collaborate with in the design phase” as one of the commandments, but no such luck.
In summary, Leigh pulled together an impressive collection of stories about people, companies, and nonprofits who were willing to put in the hard work up front to reap the benefits of stronger inferences downstream. This is an important message and one most statisticians find themselves repeating like a broken record. So, I again applaud Leigh for providing some great PR for us. However, he does sell randomization as if it buys more than it actually does, as if there is no lingering uncertainty after its use. So take in the main message about design, but read while taking a healthy dose of skepticism and critical thinking. As always, we should question conclusions that are stated as fact until we are convinced such strength in wording is warranted—and we should model healthy skepticism for our students and colleagues. Uncertainty should be part of the story, rather than missing in action. Ignoring the hazy gray area may seem innocuous in this context, especially relative to the convenience it affords, but I do not believe it is.
Critical Inference, LLC
Bozeman, MT