Comparing Possibly Misspecified Forecasts

Abstract

Recent work has emphasized the importance of evaluating estimates of a statistical functional (such as a conditional mean, quantile, or distribution) using a loss function that is consistent for the functional of interest, of which there is an infinite number. If forecasters all use correctly specified models free from estimation error, and if the information sets of competing forecasters are nested, then the ranking induced by a single consistent loss function is sufficient for the ranking by any consistent loss function. This article shows, via analytical results and realistic simulation-based analyses, that the presence of misspecified models, parameter estimation error, or nonnested information sets, leads generally to sensitivity to the choice of (consistent) loss function. Thus, rather than merely specifying the target functional, which narrows the set of relevant loss functions only to the class of loss functions consistent for that functional, forecast consumers or survey designers should specify the single specific loss function that will be used to evaluate forecasts. An application to survey forecasts of U.S. inflation illustrates the results.

Keywords:

• Bregman distance
• Consistent loss functions
• Economic forecasting
• Model misspecification
• Point forecasting
• Proper scoring rules
• Survey forecasts

ACKNOWLEDGMENTS

For helpful comments and suggestions, I am grateful to the editor (Todd Clark) and two referees, and also Tim Bollerslev, Dean Croushore, Frank Diebold, Tilmann Gneiting, Jia Li, Robert Lieli, Minchul Shin, Allan Timmermann and seminar participants at Boston College, Columbia, Duke, Penn, Princeton, St. Louis Federal Reserve, 8th French Economics Conference, NBER Summer Institute, Nordic Econometric Society Meetings, and the World Congress of the Econometric Society. Finally, I thank Beatrix Patton for compelling me to just sit quietly and think about this problem.