Critical review of the analysis of competing hypotheses technique: lessons for the intelligence community

Bibliography

• Abdi, H. “Bonferroni and Šidák Corrections for Multiple Comparisons.” Encyclopedia of Measurement and Statistics 3 (2007): 103–107.
   Google Scholar
• Bandyopadhyay, P. S., ed. Philosophy of Statistics. Oxford: North Holland, 2011.
   Google Scholar
• Carnap, R. Logical Foundations of Probability. Vol. 3. Edited by T. S. Gendler and James Hawthorne. London: Routledge and K. Paul, 1951.
   Google Scholar
• Chang, W., P. Atanasov, S. Patil, B. A. Mellers, and P. E. Tetlock. “Accountability and Adaptive Performance Under Uncertainty: A Long-Term View.” Judgment and Decision Making 12, no. 6 (2017): 610–626. doi:10.1017/S1930297500006732.
   Web of Science ®Google Scholar
• Chang, W., E. Berdini, D. R. Mandel, and P. E. Tetlock. “Restructuring Structured Analytic Techniques in Intelligence.” Intelligence & National Security 33, no. 3 (2018): 337–356. doi:10.1080/02684527.2017.1400230.
   Web of Science ®Google Scholar
• Cohen, J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, N.J: L. Erlbaum Associates, 1988.
   Google Scholar
• Convertino, G., D. Billman, J. P. M. Peter Pirolli, and J. Shrager. “The CACHE Study: Group Effects in Computer-Supported Collaborative Analysis.” Computer Supported Cooperative Work (CSCW) 17, no. 4 (2008): 353–393. doi:10.1007/s10606-008-9080-9.
   Web of Science ®Google Scholar
• Coulson, M., M. Healey, F. Fidler, and G. Cumming. “Confidence Intervals Permit, but Don’t Guarantee, Better Inference Than Statistical Significance Testing.” Frontiers in Psychology (2010): 26. doi:10.3389/fpsyg.2010.00026.
   PubMed Web of Science ®Google Scholar
• Coulthart, S. “Why Do Analysts Use Structured Analytic Techniques? An In-Depth Study of an American Intelligence Agency.” Intelligence & National Security 31, no. 7 (2016): 933–948. doi:10.1080/02684527.2016.1140327.
   Web of Science ®Google Scholar
• Coulthart, S. J. “An Evidence-Based Evaluation of 12 Core Structured Analytic Techniques.” International Journal of Intelligence & CounterIntelligence 30, no. 2 (2017): 368–391. doi:10.1080/08850607.2016.1230706.
   Web of Science ®Google Scholar
• Dhami, M. K., I. K. Belton, and D. R. Mandel. “The ‘Analysis of Competing Hypotheses’ in Intelligence Analysis.” Applied Cognitive Psychology 33, no. 6 (2019): 1080–1090. doi:10.1002/acp.3550.
   Web of Science ®Google Scholar
• Dhami, M. K., D. R. Mandel, B. A. Mellers, and P. E. Tetlock. “Improving Intelligence Analysis with Decision Science.” Perspectives on Psychological Science 10, no. 6 (2015): 753–757. doi:10.1177/1745691615598511.
   Web of Science ®Google Scholar
• Edwards, K., and E. E. Smith. “A Disconfirmation Bias in the Evaluation of Arguments.” Journal of Personality and Social Psychology 71, no. 1 (1996): 5. doi:10.1037/0022-3514.71.1.5.
   Web of Science ®Google Scholar
• Faul, F., E. Erdfelder, A. Buchner, and A.-G. Lang. “Statistical Power Analyses Using G* Power 3.1: Tests for Correlation and Regression Analyses.” Behavior Research Methods 41, no. 4 (2009): 1149–1160. doi:10.3758/BRM.41.4.1149.
   PubMed Web of Science ®Google Scholar
• Fidler, F., and J. Wilcox. “Reproducibility of Scientific Results.” In The Stanford Encyclopedia of Philosophy. The Stanford Encyclopedia of Philosophy, 2018. https://plato.stanford.edu/archives/win2018/entries/scientific-reproducibility/.
   Google Scholar
• Folker, R. D., Jr “Exploiting Structured Methodologies to Improve Qualitative Intelligence Analysis.” PhD Thesis, Masters thesis: Joint Military Intelligence College, 1999.
   Google Scholar
• Friedman, J. A. War and Chance: Assessing Uncertainty in International Politics. New York: Oxford University Press, 2019.
   Google Scholar
• Gelder, T. V. “Can We Do Better Than ACH?” AIPIO News, December 2008.
   Google Scholar
• Gleitman, H., J. J. Gross, and D. Reisberg. Psychology. 8th ed. New York: W. W. Norton & Co, 2011.
   Google Scholar
• Hart, W., D. Albarracín, A. H. Eagly, I. Brechan, M. J. Lindberg, and L. Merrill. “Feeling Validated versus Being Correct: A Meta-Analysis of Selective Exposure to Information.” Psychological Bulletin 135, no. 4 (2009, July): 555–588. doi:10.1037/a0015701.
   PubMed Web of Science ®Google Scholar
• Hawthorne, J. “Supplement to Inductive Logic: Likelihood Ratios, Likelihoodism, and the Law of Likelihood.” The Stanford Encyclopedia of Philosophy (Spring 2018 Edition), 2018. https://plato.stanford.edu/entries/logic-inductive/sup-likelihood.html.
   Google Scholar
• Heuer, R. J. Psychology of Intelligence Analysis. Washington, D.C: Center for the Study of Intelligence, 1999.
   Google Scholar
• Heuer, R. J. “The Evolution of Structured Analytic Techniques.” E National Academy of Science, 2009. https://www.e-education.psu.edu/geog885/sites/www.e-education.psu.edu.geog885/files/file/Evolution_SAT_Heuer.pdf.
   Google Scholar
• Heuer, R. J., and R. H. Pherson. “Structured Analytic Techniques: A New Approach to Analysis.” in George and Bruce (eds.) Analyzing Intelligence, 2nd ed., pp.231–48. (2014).
   Google Scholar
• Himmelstein, M., P. Atanasov, and D. V. Budescu. “Forecasting Forecaster Accuracy: Contributions of Past Performance and Individual Differences.” Judgment and Decision Making 16, no. 2 (2021): 323–362. doi:10.1017/S1930297500008597.
   Web of Science ®Google Scholar
• Howson, C., and P. Urbach. Scientific Reasoning: The Bayesian Approach. 3rd ed. Chicago: Open Court, 2006.
   Google Scholar
• “Intelligence Community Directive 203: Analytic Standards.” Office of the Director of National Intelligence, 2007.
   Google Scholar
• Jang, S. M. “Seeking Congruency or Incongruency Online?” Science Communication 36, no. 2 (2014, April): 143–167. doi:10.1177/1075547013502733.
   Web of Science ®Google Scholar
• Jones, N. “Critical Epistemology for Analysis of Competing Hypotheses.” Intelligence & National Security 33, no. 2 (2018): 273–289. doi:10.1080/02684527.2017.1395948.
   Web of Science ®Google Scholar
• Kahneman, D., and D. Lovallo. “Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking.” Management Science 39, no. 1 (1993): 17–31. doi:10.1287/mnsc.39.1.17.
   Web of Science ®Google Scholar
• Kahneman, D., and A. Tversky. “Subjective Probability: A Judgment of Representativeness.” Cognitive Psychology 3, no. 3 (1972): 430–454. doi:10.1016/0010-0285(72)90016-3.
   Web of Science ®Google Scholar
• Karvetski, C. W., and D. R. Mandel. “Coherence of Probability Judgments from Uncertain Evidence: Does ACH Help?” Judgment & Decision Making 15, no. 6 (2020): 939–958. doi:10.1017/S1930297500008159.
   Web of Science ®Google Scholar
• Karvetski, C. W., D. R. Mandel, and D. Irwin. “Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation.” Risk Analysis 40, no. 5 (2020): 1040–1057. doi:10.1111/risa.13443.
   PubMed Web of Science ®Google Scholar
• Karvetski, C. W., C. Meinel, D. T. Maxwell, L. Yunzi, B. A. Mellers, and P. E. Tetlock. “What Do Forecasting Rationales Reveal About Thinking Patterns of Top Geopolitical Forecasters?” International Journal of Forecasting 38, no. 2 April 1 (2022): 688–704. doi:10.1016/j.ijforecast.2021.09.003.
   Web of Science ®Google Scholar
• Kretz, D. R., B. J. Simpson, and C. Jacob Graham. “A Game-Based Experimental Protocol for Identifying and Overcoming Judgment Biases in Forensic Decision Analysis.” In 2012 IEEE Conference on Technologies for Homeland Security (HST), 439–444. Waltham, MA, USA: IEEE, 2012.
   Google Scholar
• Krosnick, J. A. “Improving Question Design to Maximize Reliability and Validity.” in The Palgrave Handbook of Survey Research, David L. Vannette and Jon A. Krosnick (eds), p. 95–101, Cham, Switzerland: Palgrave Macmillan. (2018).
   Google Scholar
• Kyburg, H., and C. Man Teng. Uncertain Inference. New York: Cambridge University Press, 2001.
   Google Scholar
• Lehner, P. E., L. Adelman, B. A. Cheikes, and M. J. Brown. “Confirmation Bias in Complex Analyses.” IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 38, no. 3 (2008): 584–592. doi:10.1109/TSMCA.2008.918634.
   Web of Science ®Google Scholar
• Lemay, A., and S. P. Leblanc. “Iterative Analysis of Competing Hypotheses to Overcome Cognitive Biases in Cyber Decision-Making.” Journal of Information Warfare 17, no. 2 (2018): 42–53.
   Google Scholar
• Maegherman, E., K. Ask, R. Horselenberg, and P. J. van Koppen. “Test of the Analysis of Competing Hypotheses in Legal Decision-Making.” Applied Cognitive Psychology 35, no. 1 (2021): 62–70. doi:10.1002/acp.3738.
   Web of Science ®Google Scholar
• Magnusson, K. “Understanding Correlations.” Accessed December 23, 2022. https://rpsychologist.com/correlation/.
   Google Scholar
• Makin, T. R., and J.-J. Orban de Xivry. “Ten Common Statistical Mistakes to Watch Out for When Writing or Reviewing a Manuscript.” eLife 8 (2019): e48175. doi:10.7554/eLife.48175.
   PubMed Web of Science ®Google Scholar
• Mandel, D. “Assessment and Communication of Uncertainty in Intelligence to Support Decision Making: Final Report of Research Task Group SAS-114.” NATO Science and Technology Organization, 2020.
   Google Scholar
• Mandel, D. R. “Instruction in Information Structuring Improves Bayesian Judgment in Intelligence Analysts.” Frontiers in Psychology 6 (2015): 387. doi:10.3389/fpsyg.2015.00387.
   PubMed Web of Science ®Google Scholar
• Mandel, D. R. “Can Decision Science Improve Intelligence Analysis.” In Researching National Security Intelligence: Multidisciplinary Approaches, edited by S. Coulthart, M. Landon-Murray, and D. Van Puyvelde, 117–140. Washington, D.C: Georgetown University Press, 2019.
   Google Scholar
• Mandel, D. R. “The Occasional Maverick of Analytic Tradecraft.” Intelligence & National Security 35, no. 3 (2020): 438–443. doi:10.1080/02684527.2020.1723830.
   Web of Science ®Google Scholar
• Mandel, D. R. “Intelligence, Science and the Ignorance Hypothesis” in The Academic-Practitioner Divide in Intelligence Studies, pp. 79-94, Rubén Arcos, Nicole K. Drumhiller and Mark Phythian (eds.), Landham, MD: Rowman & Littlefield Publishers, 2021.
   Google Scholar
• Mandel, D. R. “Conjectures on Science and Rationality: Commentary on Fiedler Et Al. (2022).” Decision 9, no. 3 (2022): 212–214. doi:10.1037/dec0000174.
   Web of Science ®Google Scholar
• Mandel, D. R., and A. Barnes. “Accuracy of Forecasts in Strategic Intelligence.” Proceedings of the National Academy of Sciences of the United States of America 111, no. 30 (2014, July): 10984–10989. doi:10.1073/pnas.1406138111.
   PubMed Web of Science ®Google Scholar
• Mandel, D. R., and A. Barnes. “Geopolitical Forecasting Skill in Strategic Intelligence.” Journal of Behavioral Decision Making 31, no. 1 (2018): 127–137. doi:10.1002/bdm.2055.
   Web of Science ®Google Scholar
• Mandel, D. R., and D. Irwin. “Tracking Accuracy of Strategic Intelligence Forecasts: Findings from a Long-Term Canadian Study.” Futures and Foresight Science 3, no. 3–4 (2021). doi:10.1002/ffo2.98.
   Google Scholar
• Mandel, D. R., C. W. Karvetski, and M. K. Dhami. “Boosting Intelligence Analysts’ Judgment Accuracy: What Works, What Fails?” Judgment and Decision Making 13, no. 6 (2018): 607–621. doi:10.1017/S1930297500006628.
   Web of Science ®Google Scholar
• Mandel, D. R., C. W. Karvetski, and D. Irwin. “Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation.” Open Science Framework, February 25, 2022. osf.io/ryfze.
   Google Scholar
• Marrin, S. “Intelligence Analysis: Structured Methods or Intuition?” American Intelligence Journal 25, no. 1 (2007): 7–16.
   Google Scholar
• Marrin, S. “Is Intelligence Analysis an Art or a Science?” International Journal of Intelligence & CounterIntelligence 25, no. 3 (2012): 529–545. doi:10.1080/08850607.2012.678690.
   Google Scholar
• McNeish, D. “On Using Bayesian Methods to Address Small Sample Problems.” Structural Equation Modeling: A Multidisciplinary Journal 23, no. 5 (2016): 750–773. doi:10.1080/10705511.2016.1186549.
   Web of Science ®Google Scholar
• Mellers, B., E. Stone, P. Atanasov, S. E. M. Nick Rohrbaugh, L. Ungar, M. M. Bishop, M. Horowitz, E. Merkle, and P. Tetlock. “The Psychology of Intelligence Analysis: Drivers of Prediction Accuracy in World Politics.” Journal of Experimental Psychology: Applied 21, no. 1 (2015): 1–14. doi:10.1037/xap0000040.
   PubMed Web of Science ®Google Scholar
• Murukannaiah, P. K., A. K. Kalia, P. R. Telangy, and M. P. Singh. “Resolving Goal Conflicts via Argumentation-Based Analysis of Competing Hypotheses.” In 2015 IEEE 23rd International Requirements Engineering Conference (RE), 156–165. Ottawa, ON, Canada: IEEE, 2015.
   Google Scholar
• Nola, R., and H. Sankey. Theories of Scientific Method: An Introduction. Philosophy and Science Theories of Scientific Method. Hoboken: Taylor and Francis, 2014.
   Google Scholar
• Nosek, B. A., C. R. Ebersole, A. C. DeHaven, and D. T. Mellor. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115, no. 11 (2018): 2600–2606. doi:10.1073/pnas.1708274114.
   PubMed Web of Science ®Google Scholar
• Office of the Director of National Intelligence. “Reports & Publications,” 2022. https://www.dni.gov/index.php/newsroom/reports-publications.
   Google Scholar
• Pham, M. T., and O. Travis Tae. “Preregistration is Neither Sufficient nor Necessary for Good Science.” Journal of Consumer Psychology 31, no. 1 (2021): 163–176. doi:10.1002/jcpy.1209.
   Web of Science ®Google Scholar
• Pherson, R. H., and R. J. Heuer. Structured Analytic Techniques for Intelligence Analysis. Thousand Oaks, California: Cq Press, 2020.
   Google Scholar
• Pollock, J. Nomic Probability and the Foundations of Induction. New York: Oxford University Press, 1990.
   Google Scholar
• Popper, K. The Logic of Scientific Discovery. London; New York: Routledge, 2005a.
   Google Scholar
• Popper, K. Realism and the Aim of Science. London; New York: Routledge, 2005b.
   Google Scholar
• Popper, K. Unended Quest: An Intellectual Autobiography. London; New York: Routledge Classics, 2005c.
   Google Scholar
• Reichenbach, Hans. The Theory of Probability : An Inquiry into the Logical and Mathematical Foundations of the Calculus of Probability/. 2nd ed. Hans, Berkeley: University of California Press, 1949.
   Google Scholar
• Sokal, A., and J. Bricmont. Fashionable Nonsense: Postmodern Intellectuals’ Abuse of Science. New York: Picador, 1999.
   Google Scholar
• Sprenger, J., and S. Hartmann. Bayesian Philosophy of Science. Oxford; New York: Oxford University Press, 2019.
   Google Scholar
• Stanovich, K. E., R. F. West, and M. E. Toplak. The Rationality Quotient: Toward a Test of Rational Thinking. Cambridge, MA: MIT Press, 2016.
   Google Scholar
• Stern, S. D. C., A. S. Cifu, and D. Altkorn. Symptom to Diagnosis : An Evidence-Based Guide. 4th ed. New York: McGraw-Hill Medical, 2020.
   Google Scholar
• Strevens, M. The Knowledge Machine: How Irrationality Created Modern Science. New York: Liveright Publishing Corporation, 2020.
   Google Scholar
• Stromer-Galley, J., P. Rossini, K. Kenski, B. McKernan, B. Clegg, J. Folkestad, C. Østerlund, L. Schooler, O. Boichak, and J. Canzonetta. “Flexible versus Structured Support for Reasoning: Enhancing Analytical Reasoning Through a Flexible Analytic Technique.” Intelligence & National Security 36, no. 2 (2021): 279–298. doi:10.1080/02684527.2020.1841466.
   Web of Science ®Google Scholar
• Tetlock, P. Expert Political Judgment: How Good is It? How Can We Know?. Princeton: Princeton University Press, 2005.
   Google Scholar
• Tetlock, P., and D. Gardner. Superforecasting: The Art and Science of Prediction. New York: Broadway Books, 2015. doi:10.1201/b15410-25.
   Google Scholar
• Tetlock, P. E., and R. Ned Lebow. “Poking Counterfactual Holes in Covering Laws: Cognitive Styles and Historical Reasoning.” American Political Science Review 95, no. 4 (2001): 829–843. doi:10.1017/S0003055400400043.
   Web of Science ®Google Scholar
• Tikuisis, P., and D. R. Mandel. “Is the World Deteriorating?” Global Governance 21, no. 1 (2015): 9–14. doi:10.1163/19426720-02101002.
   Web of Science ®Google Scholar
• US Congress. “Intelligence Reform and Terrorism Prevention Act of 2004,” 2004. https://www.govinfo.gov/content/pkg/PLAW-108publ458/html/PLAW-108publ458.htm.
   Google Scholar
• US Government. “A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis,” March 2009. https://www.stat.berkeley.edu/~aldous/157/Papers/Tradecraft%20Primer-apr09.pdf.
   Google Scholar
• Whitesmith, M. Cognitive Bias in Intelligence Analysis: Testing the Analysis of Competing Hypotheses Method. Edinburgh, Scotland: Edinburgh University Press, 2020.
   Google Scholar
• Whitesmith, M. “Justified True Belief Theory for Intelligence Analysis.” Intelligence & National Security 37, no. 6 (2022): 835–849. doi:10.1080/02684527.2022.2076332.
   Web of Science ®Google Scholar
• Wilcox, J. “Likelihood Neglect Bias and the Mental Simulations Approach: An Illustration Using the Old and New Monty Hall Problems,” Under review.
   Google Scholar
• Wilcox, J. Human Judgment: How Accurate is It, and How Can It Get Better?. Cham, Switzerland: Springer, 2022.
   Google Scholar
• Wilcox, J. “Credences and Trustworthiness: A Calibrationist Account.” Review of Philosophy and Psychology (forthcoming). https://osf.io/preprints/psyarxiv/vm34x.
   Google Scholar