Getting beyond analysis by anecdote: improving intelligence analysis through the use of case studies^*

Abstract

Since the 9/11 attacks critics of the American intelligence community have often complained about the lack of scientific rigor in intelligence analysis, and much of the work of the intelligence community has been described as mere ‘analysis by anecdote.’ In response, the intelligence community has made a considerable effort to increase the rigor of its analysis. But surprisingly little has been done to examine how intelligence professionals might benefit from adopting one of the most common methods used in the social sciences: case study analysis. This article argues that a greater understanding of how case studies are used by political scientists and other scholars can help improve the quality of intelligence analysis and help the intelligence community assist policymakers as they attempt to understand the threats and challenges of today’s world.

Notes

* The views expressed in this article are the author’s own, and do not necessarily represent the views of the U.S. government or the Naval Postgraduate School.

1. Medina, “The New Analysis,” 245–6.

2. Central Intelligence Agency, A Tradecraft Primer, 14.

3. One example is Hoven and Lawton, “Locally Nuanced Actionable Intelligence.”

4. For example, German, “Does Intelligence Have to be so Unintelligent?”

5. Jervis, Why Intelligence Fails, 23.

6. Shenon, The Commission.

7. Ibid., 220.

8. Medina, “The New Analysis,” 245–6. Ferris has referred to the focus on anecdote instead of analysis as “the Bloomsbury syndrome,” in his “Coming in from the Cold War,” 98.

9. Quoted in Marrin, Improving Intelligence Analysis, 43.

10. Breslin, “Digging JFK Grave Was His Honor.”

11. Byman, “Intelligence and Its Critics,” 272.

12. See for example Shelton, “The Roots of Analytic Failures.”

13. A Tradecraft Primer, 1.

14. See for example Marrin, Improving Intelligence Analysis, and Bruce, “Making Analysis More Reliable.” A useful review of the debate over the value of quantitative vs. qualitative methods for intelligence analysis is Bang, “Pitfalls in Military Quantitative Intelligence Analysis.”

15. National Research Council, Intelligence Analysis for Tomorrow.

16. Skinner, “Qualitative Analysis for the Intelligence Community.” See also Treverton and Gabbard, Assessing the Tradecraft of Intelligence Analysis, and Johnston, Analytic Culture in the U.S. Intelligence Community. On scenario analysis, see Feder, “Forecasting for Policy Making,” 120–2. I am indebted to Stephen Marrin for bringing this article to my attention in his Improving Intelligence Analysis.

17. A Tradecraft Primer. For an extensive discussion of many structured analytic techniques, see Heuer and Pherson, Structured Analytic Techniques for Intelligence Analysis.

18. Margolis, “Estimating State Instability.”

19. Ibid., 15.

20. Prunckun, Scientific Methods of Inquiry for Intelligence Analysis, 82.

21. Clark, Intelligence Analysis, 201.

22. May and Zelikow, Dealing With Dictators, 3.

23. See the web site of the Harvard Kennedy School Case Program, http://www.case.hks.harvard.edu/.

24. Joint Military Intelligence College, Experiences to Go, 1.

25. The case was written by Col. Thomas W. Shreeve, USMCR. The JMIC publication mentions the Intelligence Community Case Method Program, which contains mostly classified cases.

26. Experiences to Go, 36.

27. Ibid., 43.

28. Wood, “Persuading a President.”

29. Walton, Challenges in Intelligence Analysis.

30. Beebe and Pherson, Cases in Intelligence Analysis.

31. George and Bennett, Case Studies and Theory Development, 17. Another often-cited source on case study work is Yin, Case Study Research.

32. Gerring, “What Is a Case Study,” 342, italics in original.

33. Another research method that is also prized above qualitative methods among many American political scientists is what is known as formal modeling, or the use of formal methods. This usually involves the use of complex mathematical models and equations. There are other kinds of political and social science methods, as well, that do not necessarily fall into the categories of qualitative or quantitative, such as modeling, experiments, and war gaming.

34. Lieberthal, “The U.S. Intelligence Community,” 35–6.

35. Jervis, Why Intelligence Fails, 189.

36. Kahneman, Thinking, Fast and Slow.

37. Ibid., 245–8.

38. Brooks, “Forecasting Fox.”

39. George and Bennett, Case Studies and Theory Development, 6.

40. For a more detailed discussion, see Collier, “Understanding Process Tracing.”

41. George and Bennett, Case Studies and Theory Development, 67–72.

42. Ibid., 70.

43. Mockaitis, “The Irish Republican Army,” 332. For similar and related cautions about relying too heavily on the lessons of history, see the classic book by Neustadt and May, Thinking in Time.

44. Wohlstetter, Pearl Harbor: Warning and Decision, 1. Other more recent single-case studies include Jervis, “Reports, Politics, and Intelligence Failures,” and Riste, “Intelligence and the ‘Mindset.’”

45. Allison and Zelikow, Essence of Decision. For a discussion of the value of this book as a single case study, see Yin, Case Study Research.

46. Allison and Zelikow, Essence of Decision, 2.

47. Dahl, “Warning of Terror.”

48. Ibid., 46.

49. Gentry, “Warning Analysis.”

50. Ibid., 80–1.

51. Betts, Surprise Attack; Jervis, Why Intelligence Fails.

52. Nolan and MacEachin, Discourse, Dissent, and Strategic Surprise: The Institute for the Study of Diplomacy at Georgetown is also a source of excellent case studies written in the business school style.

53. Freeman, “A Theory of Terrorist Leadership.”

54. Ibid., 10 (emphasis in original).

55. Ibid., 11.

56. Ibid., 12–4.

57. Ibid., 15.

58. Hoffman, “The Global Terror Threat.”

59. See for example, Schneider, Gleditsch, and Carey, “Forecasting in International Relations.”

60. Gaddis, “International Relations Theory and the End of the Cold War,” 6. One method of estimating the future is through the use of scenarios, which might be considered a type of future counterfactual case study. This type of case study is beyond the scope of this article, but a useful discussion is Junio and Mahnken, “Conceiving of Future War.”

61. A classic statement is Betts, “Analysis, War, and Decision.” See also “Dealing with the Future: The Limits of Forecasting.”

62. For example, Clark, Intelligence Analysis, 207.

63. Engelhardt, “The Big Book of Empire.” For a less negative view, see Brooks, “Known Unknowns.”

64. Pillar, “Predictive Intelligence,” 27.

65. Horowitz and Tetlock, “Trending Upward.”

66. Friedman and Zeckhauser, “Assessing Uncertainty in Intelligence.”

67. A very useful study that takes a case study approach to examining intelligence community forecasts is Armstrong, et al., “The Hazards of Single-Outcome Forecasting.”

68. Pillar, “Predictive Intelligence,” 34.

69. Zegart, “No One Saw This Coming.”

70. Rock, Why Peace Breaks Out.

71. Gaddis, “International Relations Theory and the End of the Cold War,” 34–5.

72. Ibid., 37.

73. Rock, Why Peace Breaks Out, ix.

74. Ibid., 1–2.

75. I have elsewhere noted that this problem is particularly evident in the study of intelligence failures, because almost all such works study only failures. If we truly want to understand what causes failure and what might lead to success, we should attempt to compare failures with success. I make an initial effort to do that in my own Intelligence and Surprise Attack.

76. According to Rock’s definition, states whose objectives tend not to conflict in very many issues are more heterogeneous, while states which produce and export different kinds of commodities tend to be heterogeneous. States which are similar to each other in areas such as language, ideology, and form of government, are homogeneous. Rock, Why Peace Breaks Out, 12–5.

77. Ibid., 148.

78. Ibid., 149.

79. Ibid., 152.

80. Ibid., 152.

81. Ibid., 156.

82. Ibid., 157.

83. Gaddis, “How the Cold War Might End.” Gaddis himself later referred to this article as a “mostly unsuccessful effort at forecasting.” Gaddis, “International Relations Theory and the End of the Cold War,” note 192, 51.

84. Gaddis, “How the Cold War Might End,” 91.

85. Ibid., 92.

86. On the lack of imagination before the 9/11 attacks, see National Commission on Terrorist Attacks Upon the United States, The 9/11 Commission Report, 339–48.