Forecasting methods and principles: Evidence-based checklists

Abstract

Problem

How to help practitioners, academics, and decision makers use experimental research findings to substantially reduce forecast errors for all types of forecasting problems.

Methods

Findings from our review of forecasting experiments were used to identify methods and principles that lead to accurate forecasts. Cited authors were contacted to verify that summaries of their research were correct. Checklists to help forecasters and their clients undertake and commission studies that adhere to principles and use valid methods were developed. Leading researchers were asked to identify errors of omission or commission in the analyses and summaries of research findings.

Findings

Forecast accuracy can be improved by using one of 15 relatively simple evidence-based forecasting methods. One of those methods, knowledge models, provides substantial improvements in accuracy when causal knowledge is good. On the other hand, data models – developed using multiple regression, data mining, neural nets, and “big data analytics” – are unsuited for forecasting.

Originality

Three new checklists for choosing validated methods, developing knowledge models, and assessing uncertainty are presented. A fourth checklist, based on the Golden Rule of Forecasting, was improved.

Usefulness

Combining forecasts within individual methods and across different methods can reduce forecast errors by as much as 50%. Forecasts errors from currently used methods can be reduced by increasing their compliance with the principles of conservatism (Golden Rule of Forecasting) and simplicity (Occam’s Razor). Clients and other interested parties can use the checklists to determine whether forecasts were derived using evidence-based procedures and can, therefore, be trusted for making decisions. Scientists can use the checklists to devise tests of the predictive validity of their findings.

问题：如何帮助从业人员、学者和决策者使用实验研究成果，从而大幅降低各类预测问题的预测误差。

方法：我们利用预测实验回顾的成果，确定准确预测的方法和原则。联系被引作者，核实他们的研究综述是否正确。我们制定了清单，以帮助预测者和他们的客户实践、委托研究，这些研究遵循原则并使用有效方法。我们要求主要研究人员，通过分析和总结研究成果，找出遗漏误差或委托误差。

发现：这15个循证预测方法相对简单，使用其中之一，就可提高预测准确性。其中一种方法，即知识模型，在基本知识良好时，可大大提高预测准确性。另一方面，使用多元回归、数据挖掘、神经网络和“大数据分析”开发的数据模型不适合进行预测。

独创性：我们介绍了三种新清单，用于选择验证方法、开发知识模型和评估不确定性。基于预测黄金法则的第四个清单得到了改进。

实用性：组合预测法，使用个别方法以及交叉不同方法做出的预测，可减少高达50%的预测误差。若我们目前所使用的方法遵循稳健性原则（预测的黄金法则）和简单性原则（奥卡姆剃刀定律），我们可以降低它们的预测误差。客户和其他相关方可使用清单来确定预测是否是利用循证程序推导出的，若是，这些预测值得信赖，因此，我们可依此做出决定。科学家可使用清单设计实验，测试他们成果的预测效度。

Keywords:

• Combining forecasts
• data models
• decomposition
• equalizing
• expectations
• extrapolation
• knowledge models
• intentions
• Occam’s razor
• prediction intervals
• predictive validity
• regression analysis
• uncertainty

关键词:

• 组合预测
• 数据模型
• 分解
• 均衡
• 预期
• 外推法
• 知识模型
• 目的
• 奥卡姆剃刀定律
• 预测区间
• 预测效度
• 回归分析
• 不确定性

Acknowledgments

We thank our reviewers, Hal Arkes, Kay A. Armstrong, Roy Batchelor, David Corkindale, Alfred G. Cuzán, John Dawes, Robert Fildes, Paul Goodwin, Andreas Graefe, Rob Hyndman, Randall Jones, Magne Jorgensen, Spyros Makridakis, Kostas Nikolopoulos, Keith Ord, Don Peters, and Malcolm Wright. Thanks also to those who made useful suggestions: Raymond Hubbard, Frank Schmidt, Phil Stern, and Firoozeh Zarkesh. And to our editors: Harrison Beard, Amy Dai, Simone Liao, Brian Moore, Maya Mudambi, Esther Park, Scheherbano Rafay, and Lynn Selhat. Finally, we thank the authors of the papers that we cited for their substantive findings for their prompt confirmation and useful suggestions on how to best summarize their work.

Notes

1. From the 2003 documentary film, “Fog of War”.

2. The text of Franklin’s 1772 letter is available at https://onlinelibrary.wiley.com/doi/10.1002/9781118602188.app1/pdf.

3. García-Ferrer, A., “Professor Zellner: An Interview.” International Journal of Forecasting 14, 1998, 303–312.