Comparing Predictive Accuracy in the Presence of a Loss Function Shape Parameter

Abstract

We develop tests for out-of-sample forecast comparisons based on loss functions that contain shape parameters. Examples include comparisons using average utility across a range of values for the level of risk aversion, comparisons of forecast accuracy using characteristics of a portfolio return across a range of values for the portfolio weight vector, and comparisons using recently-proposed “Murphy diagrams” for classes of consistent scoring rules. An extensive Monte Carlo study verifies that our tests have good size and power properties in realistic sample sizes, particularly when compared with existing methods which break down when then number of values considered for the shape parameter grows. We present three empirical illustrations of the new test.

Keywords:

• Forecasting
• Model selection
• Nuisance parameters
• Out-of-sample testing

Acknowledgments

We thank Dick van Dijk, Erik Kole, Chen Zhou, and seminar participants at Oxford University for valuable discussions and feedback. The first author also acknowledges financial support from the Erasmus Trustfonds. All errors remain our own.

Notes

1 The returns can be obtained from the data library at http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html

Additional information

Funding

The first author also acknowledges financial support from the Erasmus Trustfonds