What are the odds? A measure of the small sample problems

Abstract

Decisions on whether to retain recent hires are often limited by small sample size. We empirically assess whether uncertainty in employee retention decisions could be significantly reduced by increasing sample size. Using a unique data set from professional tennis matches to measure small sample outcomes, we find little difference in giving three chances, relative to five chances, in determining innate ability.

Acknowledgement

We thank Anca Cotet, Patrick McLaughlin and Hillary Morgan for their helpful comments. Any mistakes are our own.

Notes

¹Given the model set up, we also do not have to worry about the underdog effect found in Harbaugh and Klumpp (2005).

²2007 is the earliest year in which publically available match by match data are available.

³A walkover is when one player withdraws from the match for an injury or other reason.

⁴This will give an accurate relative ranking. It is possible to also use the national ranking at the time of the tournament. However, national rankings are considered when tournament rankings are devised. One worry is that if some of the talent does not show up at a smaller (non-Grand Slam) tournament, the relative rankings will be different. This issue will be controlled for in the set up of the DID estimation.

⁵They do not control for playing surface or tournament characteristics because they only use Wimbledon data.

⁶This has also been tested as any player within 10 ranks, with similar results.