Evaluating the Quality of Intelligence Analysis: By What (Mis) Measure?

Abstract

Each of the criteria most frequently used to evaluate the quality of intelligence analysis has limitations and problems. When accuracy and surprise are employed as absolute standards, their use reflects unrealistic expectations of perfection and omniscience. Scholars have adjusted by exploring the use of a relative standard consisting of the ratio of success to failure, most frequently illustrated using the batting average analogy from baseball. Unfortunately even this relative standard is flawed in that there is no way to determine either what the batting average is or should be. Finally, a standard based on the decision-makers' perspective is sometimes used to evaluate the analytic product's relevance and utility. But this metric, too, has significant limitations. In the end, there is no consensus as to which is the best criteria to use in evaluating analytic quality, reflecting the lack of consensus as to what the actual purpose of intelligence analysis is or should be.

Notes

¹While the following evaluation of these criteria is drawn from and illustrated using cases from the United States' experience with intelligence, the argument is intentionally generalized to all intelligence analysis across space and time.

²George Tenet, ‘DCI Remarks on Iraq's WMD Programs as Prepared for Delivery’, 5 February 2004, <https://www.cia.gov/news-information/speeches-testimony/2004/tenet_georgetownspeech_02052004.html> (accessed 21 June 2012).

³For more on the difficulties of using accuracy as an evaluative criterion, see: Richards J. Heuer, Jr., ‘The Evolution of Structured Analytic Techniques’, presentation to the National Academy of Science, National Research Council Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security, Washington, DC, 8 December 2009, pp.4–5.

⁴A portion of this discussion on preventing surprise is drawn from Stephen Marrin, ‘Preventing Intelligence Failures by Learning from the Past’, International Journal of Intelligence and Counterintelligence 17/4 (2004) pp.655–72.

⁵For an in-depth discussion of the causes of surprise, see James J. Wirtz, ‘Theory of Surprise’ in Richard K. Betts and Thomas G. Mahnken (eds.) Paradoxes of Strategic Intelligence: Essays in Honor of Michael I. Handel (London: Frank Cass Publishers 2003) pp.101–16.

⁶See especially Gregory F. Treverton, Reshaping National Intelligence for an Age of Information (New York; Cambridge: Cambridge University Press 2001). For an earlier version of the secrets versus mysteries discussion, see Joseph S. Nye Jr. ‘Peering into the Future’, Foreign Affairs 73/4 (1994) pp.82–93.

⁷Christopher Andrew, For the President's Eyes Only: Secret Intelligence and the American Presidency from Washington to Bush (New York: HarperPerennial 1995) p.538.

⁸Richard K. Betts, ‘Analysis, War, and Decision: Why Intelligence Failures Are Inevitable’, World Politics 31/2 (1978) p.89.

⁹Richard K. Betts, Enemies of Intelligence: Knowledge and Power in American National Security (New York: Columbia University Press 2007) p.184

¹⁰Ibid., p.185

¹¹Ronald D. Garst and Max L. Gross, ‘On Becoming an Intelligence Analyst’, Defense Intelligence Journal 6/2 (1997) p.48.

¹²Betts, ‘Analysis, War, and Decision’.

¹³Klaus E. Knorr, ‘Foreign Intelligence and the Social Sciences’, Research Monograph No. 17, Center of International Studies, Woodrow Wilson School of Public and International Affairs, Princeton University, 1 June 1964, p.23.

¹⁴Betts, ‘Analysis, War, and Decision’.

¹⁵Remy Charlip, Fortunately (New York: Parents Magazine Press 1964).

¹⁶‘The Future of US Intelligence: Report Prepared for the Working Group on Intelligence Reform’, Consortium for the Study of Intelligence, 1996, p.29.

¹⁷Betts, Enemies of Intelligence, p.184.

¹⁸For example, see: John Hedley, ‘Learning from Intelligence Failures’, International Journal of Intelligence and Counterintelligence 18/3 (2005) pp.435–50; Paul R. Pillar, ‘Intelligent Design? The Unending Saga of Intelligence Reform’, Foreign Affairs, March/April 2008 (citing Betts); Robert Mandel, ‘Distortions in the Intelligence Decision-Making Process’ in Stephen J. Cimbala (ed.) Intelligence and Intelligence Policy in a Democratic Society (Dobbs Ferry, NY: Transnational Publishers, Inc. 1987) pp.69–83; Mark M. Lowenthal, ‘The Burdensome Concept of Failure’ in Alfred C. Maurer, Marion D. Turnstall, and James M. Keagle (eds.) Intelligence: Policy and Process (Boulder, CO: Westview 1985) pp.43–56; Mark M. Lowenthal, ‘Towards a Reasonable Standard for Analysis: How Right, How Often on Which Issues?’ Intelligence and National Security 23/3 (2008) pp.303–15.

¹⁹As Robert Jervis points out, ‘if we were right something like one time out of three we would be doing quite well’. Robert Jervis, ‘Improving the Intelligence Process: Informal Norms and Incentives’ in Alfred C. Maurer, Marion D. Tunstall and James M. Keagle (eds.) Intelligence: Policy and Process (Boulder, CO: Westview Press 1985) p.113.

²⁰Betts, Enemies of Intelligence, p.65.

²¹Glenn Hastedt, ‘Intelligence Estimates: NIEs vs. the Open Press in the 1958 China Straits Crisis’, International Journal of Intelligence and CounterIntelligence 23/1 (2009) pp.104–32.

²²Richard K. Betts, ‘Fixing Intelligence’, Foreign Affairs 81/1 (2002) p.59.

²³Klaus E. Knorr, ‘Failures in National Intelligence Estimates: The Case of the Cuban Missiles’, World Politics 16/3 (1964) p.460.

²⁴Betts, Enemies of Intelligence, p.186

²⁵Betts, ‘Fixing Intelligence’, p.59.

²⁶Stanley Moskowitz, ‘Intelligence in Recent Public Literature: Uncertain Shield: The U.S. Intelligence System in the Throes of Reform by Richard A. Posner’, Studies in Intelligence 50/3 (2006), <https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol50no3/Uncertain_Shield_7.htm> (accessed 21 June 2012).

²⁷Betts, Enemies of Intelligence, pp.185–6.

²⁸Richard A. Posner, Preventing Surprise Attacks: Intelligence Reform in the Wake of 9/11 (Lanham, MD: Rowman and Littlefield Publishers, Inc. 2005) p.106. Kuhns made a similar point when he answered the question ‘just how bad is it out there?’ by saying ‘the short answer is, no one knows.’ Woodrow J. Kuhns, ‘Intelligence Failures: Forecasting and the Lessons of Epistemology’ in Richard K. Betts and Thomas G. Mahnken (eds.) Paradoxes of Strategic Intelligence: Essays in Honor of Michael I. Handel (London: Frank Cass Publishers 2003), p.82.

²⁹Richard L. Russell, Sharpening Strategic Intelligence: Why the CIA Gets It Wrong and What Needs to Be Done to Get It Right (Cambridge: Cambridge University Press 2007); Richard J. Kerr, ‘The Track Record: CIA Analysis from 1950 to 2000’ in Roger Z. George and James B. Bruce (eds.) Analyzing Intelligence: Origins, Obstacles, and Innovations (Washington, DC: Georgetown University Press 2008) pp.35–54.

³⁰James G. Blight, and David A. Welch, ‘The Cuban Missile Crisis and Intelligence Performance’ in James G. Blight and David A. Welch (eds.) Intelligence and the Cuban Missile Crisis (London: Frank Cass 1998) p.189.

³¹Ibid., p.190.

³²Ibid., pp.190–1.

³³Ibid., pp.201, 192.

³⁴Ibid., p.192.

³⁵Ibid, p.193.

³⁶Ibid., pp.189, 193.

³⁷Kuhns, ‘Intelligence Failures’, p.82.

³⁸Abbot E. Smith, ‘On the Accuracy of National Intelligence Estimates’, Studies in Intelligence 13/3 (1969) p.25. As cited in Kuhns, ‘Intelligence Failures’, p.82.

³⁹Kuhns, ‘Intelligence Failures’, p.82.

⁴⁰Ibid., p.83.

⁴¹Betts, Enemies of Intelligence, p.187.

⁴²Ibid., p.188.

⁴³Stephen Marrin, ‘Intelligence Analysis and Decisionmaking: Methodological Challenges’ in Peter Gill, Stephen Marrin and Mark Phythian (eds.) Intelligence Theory: Key Questions and Debates (London: Routledge 2008) pp.131–50.

⁴⁴Stephen Marrin, ‘Intelligence Analysis Theory: Explaining and Predicting Analytic Responsibilities’, Intelligence and National Security 22/6 (2007) pp.821–46. Also see Stephen Marrin, ‘At Arm's Length or at the Elbow?: Explaining the Distance between Analysts and Decisionmakers’, International Journal of Intelligence and Counterintelligence 20/3 (2007) pp.401–14.

⁴⁵Kuhns, ‘Intelligence Failures’, p.82.

⁴⁶Robert M. Gates, ‘The CIA and American Foreign Policy’, Foreign Affairs 66/2 (1987/88) p.216.

⁴⁷Cherry-picking occurs when decision-makers take data or analysis out of context to make decisions or justify decisions previously reached. A variant of cherry-picking is backstopping which involves, according to Roger Hilsman, the intelligence analyst's ‘mechanical search for facts tending to support a policy decision that has already been made’ in order to protect the decision-maker by ‘supplying him with facts to defend his position’. Hilsman goes on to observe that decision-makers ‘seemed most pleased’ with backstopping. See Roger Hilsman, ‘Intelligence and Policy-Making in Foreign Affairs’, World Politics, April 1953, p.6.

⁴⁸Harold P. Ford, ‘The US Government's Experience with Intelligence Analysis: Pluses and Minuses’ in David Charters, A. Stuart Farson and Glenn P. Hastedt (eds.) Intelligence Analysis and Assessment (London: Frank Cass 1996), p.51.

⁴⁹Paul R. Pillar, ‘Intelligence, Policy, and the War in Iraq’, Foreign Affairs 85/2 (2006) pp.15–27.

⁵⁰Ford, ‘The US Government s Experience with Intelligence Analysis’, p.39.

⁵¹Sherman Kent, ‘Estimates and Influence’, Studies in Intelligence 12/3 (1968) pp.11–21, <https://www.cia.gov/library/center-for-the-study-of-intelligence/kent-csi/vol12i3/html/v12i3a02p_0001.htm> (accessed 21 June 2012).

⁵²Ibid.

⁵³Ford, ‘The US Government’s Experience with Intelligence Analysis’, p.50.

⁵⁴Ibid., p.50.

⁵⁵Richard H. Immerman, ‘Intelligence and Strategy: Historicizing Psychology, Policy and Politics’, Diplomatic History 32/1 (2008) p.7.

⁵⁶Medicine and medical diagnosis is frequently referenced in the intelligence literature as an analogous domain. See Stephen Marrin and Jonathan Clemente, ‘Improving Intelligence Analysis by Looking to the Medical Profession’, International Journal of Intelligence and Counterintelligence 18/4 (2005) pp.707–29; Stephen Marrin and Jonathan Clemente, ‘Modeling an Intelligence Analysis Profession on Medicine’, International Journal of Intelligence and Counterintelligence 19/4 (2006–2007) pp.642–65; and Josh Kerbel, ‘Lost for Words: The Intelligence Community's Struggle to Find Its Voice’, Parameters 38/2 (2008) pp.102–12.

⁵⁷Jerome E. Groopman, How Doctors Think (New York: Houghton Mifflin Company 2007) p.7.

⁵⁸For an example of this kind of emphasis, see: Uri Bar-Joseph and Rose McDermott, ‘Change the Analyst and Not the System: A Different Approach to Intelligence Reform’, Foreign Policy Analysis 4/2 (2008) pp.127–45.