Penalized KS method to fit data sets with power law distribution over a bounded subinterval

Abstract

We develop a variation of a Kolmogorov-Smirnov (KS) method for estimating a power law region, including its lower and upper bounds, of the probability density in a set of data which can be modelled as a continuous random sample. Our main innovation is to stabilize the estimation of the bounds of the power law region by introducing an adaptive penalization term involving the logarithmic length of the interval when minimizing the Kolmogorov-Smirnov distance between the random sample and the power law fit over various candidate intervals. We show through simulation studies that an adaptively penalized Kolmogorov-Smirnov (apKS) method improves the estimation of the power law interval on random samples from various theoretical probability distributions. Variability in the estimation of the bounds can be further reduced when the apKS method is applied to subsamples of the original random sample, and the subsample estimates are averaged to yield a final estimate.

Keywords:

• Truncated power law
• bounded power law
• power law interval
• KS
• validation
• penalization

2010 Mathematics Subject Classification:

• 62F99

Acknowledgments

PRK would like to thank the Mathematical Biosciences Institute at Ohio State University for support as a long-term visitor during its emphasis year on Stochastics in Biological Systems, during which this collaboration was initiated. We also thank Suzanne Ditlevsen for pointing out the relation of our proposed batching strategy to a technique used in spectral density estimation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work of FO was partially supported by National Science Foundation in China [grant no. 31571071]. The work of JB was partially supported by NSF award DMS-0956057.