Understanding Heavy Tails in a Bounded World or, is a Truncated Heavy Tail Heavy or Not?

Abstract

We address the important question of the extent to which random variables and vectors with truncated power tails retain the characteristic features of random variables and vectors with power tails. We define two truncation regimes, soft truncation regime and hard truncation regime, and show that, in the soft truncation regime, truncated power tails behave, in important respects, as if no truncation took place. On the other hand, in the hard truncation regime much of “heavy tailedness” is lost. We show how to estimate consistently the tail exponent when the tails are truncated, and suggest statistical tests to decide on whether the truncation is soft or hard. Finally, we apply our methods to two recent data sets arising from computer networks.

Keywords:

• Central limit theorem
• Consistency
• Heavy tails
• Hill estimator
• Regular variation
• Truncation

Mathematics Subject Classification:

• Primary 62G32, 62G10
• Secondary 60E07

ACKNOWLEDGMENTS

The authors wish to thank Dr. F. Donelson Smith of the Network Research Laboratory in the Computer Science Department at the University of North Carolina at Chapel Hill for kindly providing the data sets analyzed in Section 5. We would also like to thank the two anonymous referees for their useful comments that lead to improvement of the presentation in the article.

Research partly supported by the ARO grant W911NF-07-1-0078 at Cornell University. Gennady Samorodnitsky's research was also partly supported by a Villum Kann Rasmussen Visiting Professor Grant at the University of Copenhagen and by Otto Moensted foundation grant at Danish Technological University.