Structured Analytic Techniques: A Pragmatic Approach

Abstract

Cognitive scientists question the effectiveness of structured analytic techniques (SATs), concluding there is no evidence they reduce biases. In some cases, SAT usage might even add new biases. Yet the U.S. Intelligence Community remains committed to promoting these techniques to improve analytic tradecraft, although some evidence suggests many analysts still do not use them. Practitioners themselves often see value in SATs but doubt they substitute for expertise and intuition. The IC needs a better understanding of how SATs improve analysis, and particularly, how they help generate insight, the key value-added for intelligence products. This article sizes up the current debate and offers practical suggestions on how SATs might best be used in light of current research.

This article is related to:

The Psychology of Intelligence

DISCLOSURE STATEMENT

No potential conflict of interest was reported by the author.

Notes

1 Stephen Artner, Richard S. Girven, and James B. Bruce, “Assessing the Value of Structured Analytic Techniques in the U.S. Intelligence Community” (RAND Corporation, 2016), p. 2. Also see, “A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis” U.S. Government (March 2009). https://www.cia.gov/resources/csi/books-monographs/a-tradecraft-primer/

2 Roger Z. George, “Beyond Analytic Tradecraft,” International Journal of Intelligence and CounterIntelligence, Vol. 23, No. 2 (2015), p. 303.

3 John A. Gentry, “The ‘Professionalization’ of Intelligence Analysis: A Skeptical Perspective,” The International Journal of Intelligence and CounterIntelligence, Vol. 29 (2016), p. 648.

4 John A. Gentry, “Has the ODNI Improved U.S. Intelligence Analysis?” International Journal of Intelligence and CounterIntelligence, Vol. 28, No. 4 (2015), p. 646.

5 Stephen Marrin, “Intelligence Analysis: Structured Methods or Intuition?” American Intelligence Journal, Vol. 25, No. 1 (Summer 2007), pp. 13–14.

6 James B. Bruce and Roger George, “Professionalizing Intelligence Analysis,” Journal of Strategic Security, Vol. 8, No. 3 (2015), p. 14.

7 Welton Chang, Elissabeth Berdini, David R. Mandel, and Philip E. Tetlock, “Restructuring Structured Analytic Techniques in Intelligence,” Intelligence and National Security, Vol. 33, No. 3 (2018), pp. 344–345.

8 David R. Mandel, “Intelligence, Science, and the Ignorance Hypothesis,” in The Academic-Practitioner Divide in Intelligence Studies, edited by Rubén Arcos, Nicole K. Drumhiller, and Mark Phythian (Lantham, MD: Rowman and Littlefield, 2022), pp. 87–89.

9 David R. Mandel, Tonya L. Hendriks, and David Irwin, “Policy for Promoting Analytic Rigor in Intelligence: Professionals’ Views and Their _Psychological Correlates,” Intelligence and National Security, Vol. 37, No. 2 (2022), p. 188.

10 Rob Johnston, Analytic Culture in the U.S. Intelligence Community (Washington, DC: Center for the Study of Intelligence, 2005), p. 15.

11 Patrick McGlynn and Godfrey Garner, Intelligence Analysis Fundamentals (Boca Raton, FL: Taylor & Francis Group, 2018), p. 139.

12 Bruce E. Pease, Leading Intelligence Analysis: Lessons from the CIA’s Analytic Front Lines (Thousand Oaks, CA: CQ Press, 2020), p. 65. Also, an e-mail to the author from retired senior CIA manager Matthew Berrett (30 August 2022).

13 José-Miguel Palacios, “Assessing the Quality of Strategic Intelligence Products: Cooperation and Competition between Scholars and Practitioners,” in The Academic-Practitioner Divide in Intelligence Studies, edited by Rubén Arcos, Nicole K. Drumhiller, and Mark Phythian (Lantham, MD: Rowman and Littlefield, 2022), p. 115.

14 S. Stenslie, L. Haugom, and B. H. Vaage (eds.), Intelligence Analysis in the Digital Age (London: Taylor & Francis Group, 2021), p. 44.

15 According to Mandel and Tetlock, the idea spread, not because it worked, but because “influential professionals in the most powerful organization have endorsed it and no one wants to fall behind prevailing norms of best practices.” David E. Mandel and Philip Tetlock, “Correcting Judgment Correctives in National Security Intelligence,” Perspective, Frontiers in Psychology Vol. 9 (December 2018), p. 2.

16 Voluntary Qualtrics survey of intelligence analysts conducted on social media by the author, 23–25 August 2022 with 28 respondents.

17 Interviews with Barry Zulauf, adjunct professor at Georgetown University, and Dale Avery, adjunct professor at Embry Riddle Aeronautical University (22 August 2022). As a program director for two intelligence programs, the author has had a similar experience.

18 Allan Collins and William Ferguson, “Epistemic Forms and Epistemic Games: Structures and Strategies to Guide Inquiry,” Educational Psychologist, Vol. 28, No. 1 (1993), p. 28.

19 Richards J. Heuer, Jr., Psychology of Intelligence Analysis (Washington, DC: The Center for the Study of Intelligence, 1999), pp. 88–89.

20 Todoist, The Eisenhower Matrix, https://todoist.com/productivity-methods/eisenhower-matrix

21 Tim van Gelder, Morgan Saletta, Ashley Barnett, Tamar Primorzatz, Luke Thorburn, and Andrew Wright, “Collective View of Analytic Rigour in Intelligence: Results of an Expert Panel Process” (University of Melbourne: Hunt Lab for Intelligence Research, 30 October 2020), p. 27.

22 Heuer, Psychology of Intelligence Analysis, p. 65.

23 Kathy Hibbs Pherson and Randolph H. Pherson, Critical Thinking for Strategic Intelligence, 3rd ed. (Thousand Oaks, CA: CQ Press, 2021), p. 69.

24 Richards J. Heuer, Jr., “The Evolution of Structured Analytic Techniques” (presentation to the National Academy of Science, National Research Council Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security, Washington, DC, 8 December 2009), p. 3.

25 Gentry, “The ‘Professionalization’ of Intelligence Analysis,” p. 65. This seems to be a common interpretation of Daniel Kahneman’s Thinking, Fast and Slow (New York: FSG, 2011).

26 Stephen Artner, Richard S. Girven, and James B. Bruce, “Assessing the Value of Structured Analytic Techniques in the U.S. Intelligence Community” (RAND Corporation, 2015), pp. 9–10. Despite demonstrating positive outcomes for SAT-based intelligence products, the intelligence community did not fund a continuation of this study. Interview with Stephen Artner (27 August 2022).

27 Stephen J. Coulthart, “An Evidence-Based Evaluation of 12 Core Structured Analytic Techniques,” International Journal of Intelligence and CounterIntelligence, Vol. 30 (2017), p. 384. This is ironic given how devil’s advocacy has fallen into disuse, and ACH has undergone important criticism.

28 Randolph H. Pherson and Richards J. Heuer, Jr., Structured Analytic Techniques for Intelligence Analysis (Thousand Oaks, CA: Sage, 2021), p. 360.

29 Bruce and George, “Professionalizing Intelligence Analysis,” p. 16.

30 Pherson and Pherson, Critical Thinking for Strategic Intelligence, p. 69.

31 Robert Pool, Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary (Washington, DC: National Research Council, 2010), p. 20.

32 Mandel, “Intelligence, Science, and the Ignorance Hypothesis,” p. 84.

33 Ibid.

34 Chang et al., “Restructuring Structured Analytic Techniques in Intelligence,” p. 342.

35 Ibid., p. 344.

36 Author interview with chief executive officer of Pyrra Technologies Welton Chang (30 August 2022).

37 Heuer, The Psychology of Intelligence Analysis, pp. 108–109.

38 Mandel, Hendriks, and Irwin, “Policy for Promoting Analytic Rigor in Intelligence,” p. 188.

39 David R. Mandel and Philip E. Tetlock, “Correcting Judgment Correctives in National Security Intelligence,” Frontiers in Psychology, Vol. 9 (December 2021), p. 4.

40 Martha Whitesmith, “The Efficacy of ACH in Mitigating Serial Position Effects and Confirmation Bias in an Intelligence Analysis Scenario,” Intelligence and National Security, Vol. 34, No. 2 (2019), p. 239.

41 David R. Mandel, Christopher W. Karveski, and Mandeep K. Dhami, “Boosting Intelligence Analysts’ Judgment Accuracy: What Works, What Fails?” Judgment and Decision Making, Vol. 13, No. 6 (2018), p. 618.

42 Thomas Fingar, Reducing Uncertainty: Intelligence Analysis and National Security (Stanford, CA: Stanford University Press, 2011), p. 33.

43 Cited in Adrian Wolfberg, In Pursuit of Insight: The Everyday Work of Intelligence Analysts Who Solve Real World Problems (Washington, DC: National Intelligence University, 2022), pp. 9–10.

44 Gary Klein and Andrea Jarosz, “A Naturalistic Study of Insight,” Journal of Cognitive Engineering and Decision Making, Vol. 5, No. 4 (2011), p. 336.

45 Ibid., p. 349.

46 Wolfberg, In Pursuit of Insight, p. 63.

47 Ibid., p. 40.

48 Chang et al., “Restructuring Structured Analytic Techniques in Intelligence,” p. 349.

49 Author interview with retired senior intelligence officer and author Robert Clark (26 August 2022).

50 Pease, Leading Intelligence Analysis, p. 65.

51 Ibid., p. 64.

52 George, “Beyond Analytic Tradecraft,” p. 303.

53 Author interview with Robert Clark (26 August 2022).

54 Stephen Marrin, “Analytic Objectivity and Science: Evaluating the US Intelligence Community’s Approach to Applied Epistemology,” Intelligence and National Security, Vol. 35, No. 3 (2020), p. 361.

55 Author interview with retired senior CIA officer Bruce Pease (19 August 2022).

56 E-mail to author from retired senior CIA analyst Barbara Sude (1 September 2022).

57 National Research Council of the National Academies, Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security, Intelligence Analysis for Tomorrow: Advances from the Behavioral and Social Sciences (National Academies Press, 2011), p. 35. DOI: https://doi.org/10.17226/13040. https://nap.nationalacademies.org/catalog/13040/intelligence-analysis-for-tomorrow-advances-from-the-behavioral-and-social

58 Author interview with Dale Avery, former national intelligence officer for the Western Hemisphere and director of the ODNI’s Office of Analytic Integrity and Standards (22 August 2022).

59 Author interview with Barry Zulauf (22 August 2022).

60 ODNI, “IC Assessments of COVID-19 Origins,” 29 October 2021, https://www.dni.gov/index.php/newsroom/reports-publications/reports-publications-2021/item/2263-declassified-assessment-on-covid-19-origins, p. 15.

61 Author interview with Robert Clark (26 August 2022).

62 Pease, Leading Intelligence Analysis, pp. 19–20.

63 Author interview with former senior CIA manager Bruce Pease (19 August 2022). A point also emphasized by Dale Avery.

64 Author interview with former director of the CIA’s Kent School, Joseph Gartin (19 August 2022).

65 Author interview with retired CIA senior analyst Harry Yeide (23 August 2022).

66 For a description of the 1990 Yugoslavia NIE, see Robert M. Clark, Intelligence Analysis: A Target-Centric Approach (7th ed.) (Washington, DC: CQ Press 2023), pp. 444–445.

67 See https://www.dni.gov/files/ODNI/documents/assessments/GlobalTrends_2040.pdf. The drawbacks to this approach can be discovered by rereading the National Intelligence Council’s 1997 Global Trends 2010 report, which, among other things, missed the rise of China and international jihad (https://www.dni.gov/index.php/who-we-are/organizations/mission-integration/nic/nic-related-menus/nic-related-content/global-trends-2010).

68 Michael J. Ard, “Lessons Learned for the Private-Sector Intelligence Analyst,” in The Academic-Practitioner Divide in Intelligence Studies, edited by Rubén Arcos, Nicole K. Drumhiller, and Mark Phythian (Lantham, MD: Rowman and Littlefield, 2022), p. 138.

69 E-mail to author from former CIA senior analyst Lincoln Krause (29 August 2022).

70 Author interview with Dale Avery (22 August 2022).

71 James B. Bruce, “Making Analysis More Reliable: Why Epistemology Matters to Intelligence,” in Analyzing Intelligence: National Security Practitioners’ Perspective (2nd ed.), edited by Roger Z. George and James B. Bruce (Washington, DC: Georgetown University Press, 2014), p. 149.

72 Author interview with Joseph Gartin (19 August 2022).

73 Micah Zenko, Red Team: How to Succeed by Thinking Like the Enemy (New York: Basic Books, 2015), p. 237.

74 Zenko, Red Team, p. 6.

75 E-mail to author from retired senior CIA manager Matthew Berrett (30 August 2022).

76 Paul C Avey and Michael C. Desch, “What Do Policymakers Want from Us? Results of a Survey of Current and Former Senior National Security Decision Makers,” International Studies Quarterly (2014), pp. 58, 238, 242.

77 Palacios, “Assessing the Quality of Strategic Intelligence Products,” pp. 120–121.

78 For some ideas on improving forecast accuracy, see Mandel and Tetlock, “Correcting Judgment Correctives in National Security Intelligence,” p. 4.

79 Heuer, Psychology of Intelligence Analysis, p. 111, note 88; Welton Cheng believed Heuer’s book needed updating. Author’s interview (30 August 2022).

80 See Gary Klein, “A Naturalistic Decisionmaking Perspective on Studying Intuitive Decision Making,” Journal of Applied Research in Memory and Cognition, Vol 4 (2015), p. 164.

81 Gary Klein, “Critical Thoughts about Critical Thinking,” Theoretical Issues in Ergonomics Science, Vol. 12, No. 3 (2011), p. 215.

82 Steven Pinker, Rationality (New York: Viking, 2021), p. 36.

83 Jonathan Rauch, The Constitution of Knowledge (Washington, DC: Brookings University Press, 2021), p. 101.

84 George, “Beyond Analytic Tradecraft,” p. 303.

85 E-mail to author from Matthew Berrett (1 September 2022).

Additional information

Notes on contributors

Michael J. Ard

Michael J. Ard, Ph.D., is the Program Director for Intelligence Analysis at Johns Hopkins University. He served as an Intelligence Analyst with the Central Intelligence Agency, which included three years as Deputy National Intelligence Officer for the Western Hemisphere and in the private sector in corporate security. A former naval officer, Ard earned his B.A. in history with honors from the College of William and Mary and his M.A. and Ph.D. in foreign affairs from the University of Virginia. He is the author of An Eternal Struggle: How the National Action Party Transformed Mexican Politics (Praeger, 2003). The author can be contacted at [email protected].