Balancing Expert Opinion and Historical Data: The Case of Baseball Umpires

Many decisions benefit from situations where there exist both ample expert opinion and historical data. In cost modeling these may include the costs of software development, the learning curve rates for specific manufacturing tasks, and the unit rate costs of operating certain products. When making forecasts we are often faced with the decision to base our estimates on either expert opinion or historical data. When these two perspectives converge, we have high confidence in the estimate. The more interesting case is when they contradict. This is where the estimator needs to dig deeper in order to determine the sources of inconsistencies.

Cost modelers are not the only ones who struggle with deciding whether to trust experts or data. Data scientists are increasingly dealing with this duality especially in the context of professional sports where expert opinion is associated with the traditional viewpoint and data-driven decision making is associated with a more modern approach. In the United States, professional sports teams are increasingly using analytics to optimize their athletes’ performance as well as their business operations (Pelton, 2015). But the culture of professional sports still depends heavily on experience and gut feel.

The case of baseball umpires provides a good example of expert opinion being preferred over historical data. In professional baseball, the umpire’s job is to determine whether the ball passed the strike zone¹ or not. If the batter does not swing it is left to the umpire’s expert judgement to identify whether the pitch was a ball or a strike. The strike zone is defined in the official rules of baseball and are not subject to interpretation, however, the implementation of measuring said strike zone is entirely left to human judgement. Even more challenging is that the decision must be made in a matter of seconds under extreme pressure.

Chen, Moskowitz, and Shue (2016) analyzed baseball umpire data using the PITCHf/x system that tracks the actual location of each pitch using multiple cameras. By comparing the umpire’s decision to the actual placement of the ball relative to the strike zone they determined that, during the 2008 to 2016 seasons which included 127 different umpires calling over 3.5 million pitches, umpires were correct only part of the time as shown in Table 1.

TABLE 1 Accuracy of MLB umpires by pitch type from 2008–2012 (Chen et al., 2016)

Pitch type (N = 1,500,000)	Accuracy (%)
Obvious pitches well within or outside the strike zone	99
All types of pitches	86.6
Ambiguous pitches close to the edge of the strike zone	60

If baseball umpires are getting one out of every eight ball/strike calls wrong, this adds up to more than 30,000 mistakes a year. In most industries, and even other professional sports leagues, this would be unacceptable but baseball traditionalists are hesitant to adopt new technologies that remove the human element from the game.

In order to compare what experts think and what the data say, it is important to define the boundaries, often blurry, that separate them. Experts are heavily influenced by first-hand experience, which may be affected by sampling biases. In other words, it would be impossible for an expert to experience all possible scenarios to leverage in their decision-making situations. Experts may further embed their own biases into the results through convenience sampling (projects that are easy to reach) or confirmation bias (projects that support one’s own views). Nevertheless, experts are able to filter relevant experiences that may apply to the project being estimated and are keen to identify context-specific attributes that make a project unique.

Cost models, which are based on empirical evidence, are perceived to be a more reliable method of estimation because of their formal and repeatable processes. These models are built on historical data and rigorously validated prior to acceptance. However, the data contain subjective assessments in their own right and are based on retrospective evaluations of projects, which may be incomplete. Additionally, data-driven models are limited in their ability to adapt to a variety of situations and are inherently deterministic, that is, the same inputs will always generate the same outputs. These causal relationships may not mirror the real world but are nevertheless the go-to method for many estimators.

The comparison between expert-based and data-based estimation is further articulated by Jørgensen and Boehm (2009) who approach the topic from diametrically opposing ends. Despite their diverse perspectives they agree on the following:

• Neither method is perfect.

• Making a one-size-fits-all decision on using models versus experts in all situations does not appear to be a good idea.

• Each methods has its strengths, which can work to an estimator’s advantage.

In fact, the combination of expert judgement and historical data has been formalized using Bayesian Inference by researchers in software engineering (Chulani, Boehm, & Steece, 1999) and ecology (Choy, O’Leary, & Mengersen, 2009). Bayesian Inference is a method of statistical inference in which Bayes’ theorem is used to update the probability for a hypothesis as more evidence or information becomes available. In the case of cost estimation, expert opinion serves as the a priori value whereas historical data serve as the a posteriori update when data become available. When combined, the prior information provided by the experts and the sample information that is obtained for completed projects provide a powerful approach that leverages their relative strengths. Ignoring the ability to leverage both updated technology and the wisdom accrued by professional umpires is similar to ignoring the impact of Bayesian Inference in mathematical modeling—it may be the difference between an accurate estimate and your team earning the bid to the pennant race.

Cost estimators may not face the same challenges that professional umpires do. Cost estimates are typically done over several workdays while umpires have tens of thousands of fans criticizing their split-second decisions. Fortunately, the cost estimating profession is based on evidence-based decision making and statistical principles, which results in a heavy use of data-driven models. Nevertheless, we must take better advantage of expert opinion because it provides relevant knowledge that formal models might miss.

Additional information

Notes on contributors

Ricardo Valerdi

Ricardo Valerdi is an Associate Professor of Systems & Industrial Engineering and Director of the Sports Management Program at the University of Arizona. He is a Visiting Fellow of the Royal Academy of Engineering (UK) and a member of the Mexican Academy of Engineering. He obtained his Ph.D. from the University of Southern California.

Notes

1. Per Major League Baseball’s “Official Baseball Rules” 2014 Edition, Rule 2.00, “The STRIKE ZONE is that area over home plate the upper limit of which is a horizontal line at the midpoint between the top of the shoulders and the top of the uniform pants, and the lower level is a line at the hollow beneath the kneecap. The Strike Zone shall be determined from the batter’s stance as the batter is prepared to swing at a pitched ball.”