The Role of Expert Judgment in Statistical Inference and Evidence-Based Decision-Making

References

• Abrams, K. R., Spiegelhalter, D., and Myles, J. P. (2004), Bayesian Approaches to Clinical Trials and Health Care, New York: Wiley.
   Google Scholar
• Anderson, H., and Hepburn, B. (2016), “Scientific Method,” available at https://plato.stanford.edu/archives/sum2016/entries/scientific-method.
   Google Scholar
• Anscombe, F. J., and Aumann, R. J. (1963), “A Definition of Subjective Probability,” The Annals of Mathematical Statistics, 34, 199–205. DOI: 10.1214/aoms/1177704255.
   Google Scholar
• Berry, S. M., Carlin, B. P., Lee, J., and Müller, P. (2010), Bayesian Adaptive Methods for Clinical Trials, Boca Raton, FL: Chapman&Hall/CRC Press.
   Google Scholar
• Bradley, R., Bell, J., Gauld, J., Lilly, A., Jordan, C., Higgins, A., and Milne, R. (2005), “UK Soil Database for Modelling Soil Carbon Fluxes and Land Use for the National Carbon Dioxide Inventory,” Report to Defra Project SP0511, Defra, London.
   Google Scholar
• Brownstein, N. C. (2018), “Perspective from the Literature on the Role of Expert Judgment in Scientific and Statistical Research and Practice,” arXiv no. 1809.04721.
   Google Scholar
• Cairney, P., and Oliver, K. (2017), “Evidence-Based Policymaking Is Not Like Evidence-Based Medicine, So How Far Should You Go to Bridge the Divide Between Evidence and Policy?,” Health Research Policy and Systems, 15, 35.
   PubMed Web of Science ®Google Scholar
• Carlin, B. P., Chaloner, K., Church, T., Louis, T. A., and Matts, J. P. (1993), “Bayesian Approaches for Monitoring Clinical Trials, With an Application to Toxoplasmic Encephalitis Prophylaxis,” The Statistician, 42, 355–367. DOI: 10.2307/2348470.
   Google Scholar
• Center for Open Science (2018), “Registered Reports,” available at https://cos.io/rr/.
   Google Scholar
• Chaloner, K., Church, T., Louis, T. A., and Matts, J. P. (1993), “Graphical Elicitation of a Prior Distribution for a Clinical Trial,” The Statistician, 42, 341–353. DOI: 10.2307/2348469.
   Google Scholar
• Collins, F. S., and Tabak, L. A. (2014), “NIH Plans to Enhance Reproducibility,” Nature, 505, 612–613. DOI: 10.1038/505612a.
   PubMed Web of Science ®Google Scholar
• Cox, D. R. (1972), “Regression Models and Life Tables” (with discussion), Journal of the Royal Statistical Society, Series B, 34, 187–220. DOI: 10.1111/j.2517-6161.1972.tb00899.x.
   Web of Science ®Google Scholar
• Cripps, E., O’Hagan, A., and Quaife, T. (2013), “Quantifying Uncertainty in Remotely Sensed Land Cover Maps,” Stochastic Environmental Research and Risk Assessment, 27, 1239–1251. DOI: 10.1007/s00477-012-0660-3.
   Web of Science ®Google Scholar
• Curtis, A., Worley, S., Adamson, P., Chung, E., Niazi, I., Sherfesee, L., Shinn, T., and Sutton, M. (2013), “Biventricular Pacing for Atrioventricular Block and Systolic Dysfunction,” New England Journal of Medicine, 368, 1585–1593. DOI: 10.1056/NEJMoa1210356.
   PubMed Web of Science ®Google Scholar
• Easterbrook, P. J., Gopalan, R., Berlin, J., and Matthews, D. R. (1991), “Publication Bias in Clinical Research,” The Lancet, 337, 867–872.
   PubMed Web of Science ®Google Scholar
• Ellenberg, S. S., Fleming, T. R., and DeMets, D. L. (2003), Data Monitoring Committees in Clinical Trials: A Practical Perspective, Chichester: Wiley.
   Google Scholar
• Finfer, S., and Bellomo, R. (2009), “Why Publish Statistical Analysis Plans,” Critical Care and Resuscitation, 11, 5–6.
   PubMed Web of Science ®Google Scholar
• Franco, A., Malhotra, N., and Simonovits, G. (2014), “Publication Bias in the Social Sciences: Unlocking the File Drawer,” Science, 345, 1502–1505. DOI: 10.1126/science.1255484.
   PubMed Web of Science ®Google Scholar
• Fuller, R., Smith, G., Sanderson, J., Hill, R., Thomson, A., Cox, R., Brown, N., Clarke, R., Rothery, P., and Gerard, F. (2002), “Countryside Survey 2000 Module 7. Land Cover Map 2000,” Final Report, CSLCM/Final.
   Google Scholar
• Garland, T., Jr. (2016), “Scientific Method as an Ongoing Process,” available at https://en.wikipedia.org/w/index.php?title=Scientific_method&oldid=822947033.
   Google Scholar
• Gelman, A., and Hennig, C. (2017), “Beyond Subjective and Objective in Statistics,” Journal of the Royal Statistical Society, Series A, 180, 967–1033. DOI: 10.1111/rssa.12276.
   Web of Science ®Google Scholar
• Gibson, E. W. (2017), “Leadership in Statistics: Increasing Our Value and Visibility,” The American Statistician, DOI: 10.1080/00031305.2017.1336484.
   Web of Science ®Google Scholar
• Gosling, J. P., and O’Hagan, A. (2006), “Understanding the Uncertainty in the Biospheric Carbon Flux for England and Wales,” Technical Report 567/06, University of Sheffield, available at http://tonyohagan.co.uk/academic/pdf/UUCF.pdf.
   Google Scholar
• Grice, J. W.(2017), “Comment on Locascio’s Results Blind Manuscript Evaluation Proposal,” Basic and Applied Social Psychology, 39, 254–255. DOI: 10.1080/01973533.2017.1352505.
   Web of Science ®Google Scholar
• Haines-Young, R., Barr, C., Black, H., Briggs, D., Bunce, R., Clarke, R., Cooper, A., Dawson, F., Firbank, L., Fuller, R., Furse, M. T., Gillespie, M. K., Hill, R., Hornung, M., Howard, D. C., McCann, T., Morecroft, M. D., Petit, S., Sier, A. R. J., Smart, S. M., Smith, G. M., Stott, A. P., Stuart, R. C., and Watkins, J. W. (2000), “Accounting for Nature: Assessing Habitats in the UK Countryside,” Natural Environment Research Council and Centre for Ecology and Hydrology, Department of the Environment, Transport and the Regions, available at https://trove.nla.gov.au/work/17604300.
   Google Scholar
• Harris, K., O’Hagan, A., and Quegan, S. (2010), “The Impact of Satellite-Derived Land Cover Uncertainty on Carbon Cycle Calculations,” available at tonyohagan.co.uk/academic/pdf/CTCDPaper_v6.pdf.
   Google Scholar
• Hawkins, D. M.(2004), “The Problem of Overfitting,” Journal of Chemical Information and Computer Sciences, 44, 1–12. DOI: 10.1021/ci0342472.
   PubMedGoogle Scholar
• Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., and Jennions, M. D. (2015), “The Extent and Consequences of P-Hacking in Science,” PLoS Biology, 13, e1002106. DOI: 10.1371/journal.pbio.1002106.
   PubMed Web of Science ®Google Scholar
• HM Treasury (2015), “The Aqua Book: Guidance on Producing Quality Analysis for Government,” HM Government, London, UK, nd https://www.gov.uk/government/publications/theaqua-book-guidance-on-producing-quality-analysis-for-government (accessed July 10, 2017).
   Google Scholar
• Hyman, M. R.(2017), “Can ‘Results Blind Manuscript Evaluation’ Assuage ‘Publication Bias’?,” Basic and Applied Social Psychology, 39, 247–251. DOI: 10.1080/01973533.2017.1350581.
   Web of Science ®Google Scholar
• Jacobson, M., Besch, C., Child, C., Hafner, R., Matts, J., Muth, K., Wentworth, D., Neaton, J., Abrams, D., Rimland, D., and Perez, G. (1994), “Primary Prophylaxis With Pyrimethamine for Toxoplasmic Encephalitis in Patients With Advanced Human Immunodeficiency Virus Disease: Results of a Randomized Trial,” The Journal of Infectious Diseases, 169, 384–394. DOI: 10.1093/infdis/169.2.384.
   PubMed Web of Science ®Google Scholar
• Kennedy, M., Anderson, C., O’Hagan, A., Lomas, M., Woodward, I., Gosling, J. P., and Heinemeyer, A. (2008), “Quantifying Uncertainty in the Biospheric Carbon Flux for England and Wales,” Journal of the Royal Statistical Society, Series A, 171, 109–135.
   Web of Science ®Google Scholar
• Kerr, N. L. (1998), “Harking: Hypothesizing After the Results Are Known,” Personality and Social Psychology Review, 2, 196–217, PMID: 15647155. DOI: 10.1207/s15327957pspr0203_4.
   PubMedGoogle Scholar
• Kynn, M. (2008), “The ‘Heuristics and Biases’ Bias in Expert Elicitation,” Journal of the Royal Statistical Society, Series A, 171, 239–264.
   Web of Science ®Google Scholar
• LaVange, L. M. (2014), “The Role of Statistics in Regulatory Decision Making,” Therapeutic Innovation & Regulatory Science, 48, 10–19. DOI: 10.1177/2168479013514418.
   PubMed Web of Science ®Google Scholar
• Lindsay, R. M., and Ehrenberg, A. S. (1993), “The Design of Replicated Studies,” The American Statistician, 47, 217–228. DOI: 10.2307/2684982.
   Web of Science ®Google Scholar
• Locascio, J. J. (2017a), “Rejoinder to Responses on ‘Results-Blind Publishing’,” Basic and Applied Social Psychology, 39, 258–261. DOI: 10.1080/01973533.2017.1356305.
   PubMed Web of Science ®Google Scholar
• Locascio, J. J. (2017b), “Results Blind Science Publishing,” Basic and Applied Social Psychology, 39, 239–246. DOI: 10.1080/01973533.2017.1336093.
   PubMed Web of Science ®Google Scholar
• Marks, M. (2017), “Commentary on Locascio,” Basic and Applied Social Psychology, 39, 252–253. DOI: 10.1080/01973533.2017.1350580.
   Web of Science ®Google Scholar
• McNutt, M. (2014), “Journals Unite for Reproducibility,” Science, 346, 679–679.
   Web of Science ®Google Scholar
• National Academies of Sciences, Engineering, and Medicine, Division of Behavioral and Social Sciences and Education, Committee on National Statistics, Panel on Improving Federal Statistics for Policy and Social Science Research Using Multiple Data Sources and State-of-the-Art Estimation Methods, Harris-Kojetin, B. A., Groves, R. M., eds. (2017), Federal Statistics, Multiple Data Sources, and Privacy Protection: Next Steps, Washington, DC: National Academies Press.
   Google Scholar
• National Research Council, Division on Engineering and Physical Sciences, Board on Mathematical Sciences and Their Applications, and Committee on Mathematical Foundations of Verification, Validation, and Uncertainty Quantification (2012a), Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification, Washington, DC: National Academies Press.
   Google Scholar
• National Research Council, Division of Behavioral and Social Sciences and Education, Division on Engineering and Physical Sciences, Committee on National Statistics, Computer Science and Telecommunications Board, and Panel on Communicating National Science Foundation Science and Engineering Information to Data Users (2012b), Communicating Science and Engineering Data in the Information Age, Washington, DC: National Academies Press.
   Google Scholar
• NICE (2012), “The Guidelines Manual: Process and Methods,” available at https://www.nice.org.uk/process/pmg6/.
   Google Scholar
• Nussbaum, B. (2017), “Introductory Remarks: JSM 2017 Presidential Invited Address,” available at https://ww2.amstat.org/meetings/jsm/2017/webcasts/index.cfm.
   Google Scholar
• Oakley, J. (2016), “Introduction to Uncertainty Quantification and Gaussian Processes,” available at http://gpss.cc/gpuqss16/slides/oakley.pdf.
   Google Scholar
• O’Brien, P. C., and Fleming, T. R. (1979), “A Multiple Testing Procedure for Clinical Trials,” Biometrics, 35, 549–556.
   PubMed Web of Science ®Google Scholar
• O’Hagan, A. (2006), “Bayesian Analysis of Computer Code Outputs: A Tutorial,” Reliability Engineering & System Safety, 91, 1290–1300. DOI: 10.1016/j.ress.2005.11.025.
   Web of Science ®Google Scholar
• O’Hagan, A. (2012), “Probabilistic Uncertainty Specification: Overview, Elaboration Techniques and Their Application to a Mechanistic Model of Carbon Flux,” Environmental Modelling & Software, 36, 35–48. DOI: 10.1016/j.envsoft.2011.03.003.
   Web of Science ®Google Scholar
• O’Hagan, A. (2018), “Expert Knowledge Elicitation: Subjective, But Scientific,” [submitted].
   Google Scholar
• Oliver, K. A., and de Vocht, F. (2017), “Defining ‘Evidence’ In Public Health: A Survey of Policymakers’ Uses and Preferences,” European Journal of Public Health, 27, 112–117. DOI: 10.1093/eurpub/ckv082.
   PubMed Web of Science ®Google Scholar
• Oliver, K., Lorenc, T., and Innvaer, S. (2014), “New Directions in Evidence-Based Policy Research: A Critical Analysis of the Literature,” Health Research Policy and Systems, 12, 34, DOI: 10.1186/1478-4505-12-34.
   PubMed Web of Science ®Google Scholar
• Ott, M. G. (1991), “Importance of the Study Protocol in Epidemiologic Research,” Journal of Occupational Medicine, 33, 1236–1239.
   PubMedGoogle Scholar
• Oxford University Press (2018a), “Judgment,” available at https://en.oxforddictionaries.com/definition/judgment.
   Google Scholar
• Oxford University Press (2018b), “Knowledge,” available at https://en.oxforddictionaries.com/definition/knowledge.
   Google Scholar
• Oxford University Press (2018c), “Opinion,” available at https://en.oxforddictionaries.com/definition/opinion.
   Google Scholar
• Oxford University Press (2018d), “Scientific Method,” available at https://en.oxforddictionaries.com/definition/scientific_method.
   Google Scholar
• Pocock, S. J. (2006), “Current Controversies in Data Monitoring for Clinical Trials,” Clinical Trials, 3, 513–521. DOI: 10.1177/1740774506073467.
   PubMed Web of Science ®Google Scholar
• Shih, J. (1995), “Sample Size Calculation for Complex Clinical Trials With Survival Endpoints,” Controlled Clinical Trials, 16, 395–407. DOI: 10.1016/S0197-2456(95)00132-8.
   PubMedGoogle Scholar
• Wasserstein, R. L., and Lazar, N. A. (2016), “The ASA’s Statement on p-Values: Context, Process, and Purpose,” The American Statistician, 70, 129–133. DOI: 10.1080/00031305.2016.1154108.
   Web of Science ®Google Scholar
• Woodward, F. I., and Lomas, M. R. (2004), “Vegetation Dynamics—Simulating Responses to Climatic Change,” Biological Reviews, 79, 643–670. DOI: 10.1017/S1464793103006419.
   PubMed Web of Science ®Google Scholar