Towards a better framework for estimative intelligence – addressing quality through a systematic approach to uncertainty handling

ABSTRACT

The analytic standards governing the production of intelligence are outlined in a number of Intelligence Community Directives (ICDs). In this paper, we are concerned with ICDs 203, 206 and 208 and, in particular, how these relate to the handling of uncertainty in estimative intelligence. An inductive thematic analysis is employed which identifies several recurring themes. In addition, a conceptual map is developed which highlights relationships and the level of inter-connectedness between the standards. Requirements for improved operationalization of uncertainty handling are also discussed. The question of analytic feasibility is then examined in relation to the five themes extracted from the earlier analysis. The paper concludes that a new framework for uncertainty handling is required and suggests that such a framework should contain a process to assess analytic feasibility from the outset of a study.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Johnson, ‘Intelligence shocks’.

2. Warner and McDonald, US Intelligence Community Reform Studies.

3. U.S. Congress, Intelligence reform and terrorism … ; U.S. WMD Commission, Commission on the Intelligence Capabilities; US H. Rept., 107–792- Joint Inquiry.

4. ODNI, ICD- 203 (2007 and 2015).

5. NATO, AJP-2; NATO, AJP-2.1; US JCoS, Joint Pub. 2–0.

6. ODNI, ICD- 208 (2008 and 2017).

7. Pigg, ‘Common Analytic Standards’; Immerman, ‘Transforming Analysis’.

8. Gentry, ‘Has the ODNI Improved … ’; Marchio, ‘If the Weatherman Can … ’; Marchio, ‘Analytic Tradecraft’; Marchio, ‘Review of National Intelligence’; Marcoci et al., ‘ODNI as an analytic ombudsman’.

9. Marchio, ‘Analytic Tradecraft’; Marchio, ‘If the Weatherman Can … ’ −32.

10. Borek, ‘Developing an Understanding’; Borek, ‘Analytic Tradecraft in the U.S’.; Mangio & Wilkinson, ‘Intelligence Analysis’.

11. Gentry, ‘Has the ODNI Improved’ −651.

12. McEachin, ‘The tradecraft of Analysis’; Davis, ‘CIA- A Compendium … ’.

13. Marrin, ‘Analytic objectivity and science’ −1.

14. Gack, ‘Analytic Standards’; Marcoci et al., ‘ODNI as an analytic ombudsman’.

15. ODNI, ODNI Rating Scale.

16. US Air Force, Air Force Handbook 14–133.

17. U.S. Congress, Intelligence reform and terrorism.

18. CIGIE, Quality Standards for Inspection; Gentry, ‘Has the ODNI Improved’.

19. U.S. DoD, Unclassified Report; U.S. DoD, Evaluation of Combatant Command; U.S. Senate, Report of the U.S. Senate.

20. Braun & Clarke, ‘Using thematic analysis’; Vaismoradi et al., ‘Content analysis and thematic analysis’.

21. Ibid.

22. Gentry, ‘Has the ODNI Improved … ’.

23. Marshall, ‘The Joker is Wild’.

24. Heuer & Pherson, Structured Analytic Techniques; US Government, A Tradecraft Primer.

25. Canton, ‘The Active Management of Uncertainty’; Muller, ‘Intelligence analysis’.

26. Bruce, Making analysis more reliable … − 181.

27. See Appendix 01, themes 01–04.

28. Butler et al., Review of Intelligence; U.S. Senate, Report on the U.S. Intelligence Community.

29. Cooper, Curing Analytic Pathologies; Phythian, ‘The Perfect Intelligence Failure?’; Wirtz, ‘The Art of the Intelligence Autopsy’.

30. e.g. by Peterson, Appropriate factors to consider; Wozny, Optimizing Communication.

31. Irwin & Mandel, Chapter 18.

32. Wheaton, ‘Evaluating Intelligence’ and ‘The Revolution Begins’.

33. Müller-Wille, ‘Intelligence and Democratic Accountability’; Moran, ‘The Pursuit of Intelligence History’.

34. NATO RTO, NATO Guide (Client-oriented Volume).

35. Marrin, ‘Intelligence Analysis’; Adamski, ‘Enhancing the ODNI Analytic Standards’.

36. Heuer & Pherson, Structured Analytic Techniques.

37. Dhami et al., «The analysis of competing hypothesis”; Chang et al., ‘Restructuring structured analytic techniques … ’.

38. ODNI, ICD- 208.

39. Cardillo, ‘A Cultural Evolution’ 2.

40. e.g. NATO, AJP-2.1; UK MOD, JDP 2–00; US DOD, Joint Publication 2–01.

41. Rojas, Masters of Analytical tradecraft- 11.

42. NATO RTO, NATO Guide (Analyst-oriented Volume)- xii.

43. Michealson, MORS Workshop- 7.

44. Mandel et al., ‘Policy for promoting analytic rigor … ’.

45. Barnett et al., ‘Analytic rigour in intelligence’.

46. Marchio, ‘If the Weatherman Can … ’; Marchio, ‘Analytic Tradecraft … ’; Marrin, ‘Evaluating the Quality … ’; Wheaton, ‘Evaluating Intelligence … ’.

47. Marrin, ‘Evaluating the Quality … ’.

48. Novak & Cañas, The theory underlying Concept Maps.

49. Marcoci et al, ‘Better together: Reliable application of the Post-9/11 and Post-Iraq … ’.

50. Isaksen & McNaught, ‘Uncertainty handling in estimative intelligence’.

51. Kent, ‘Words of Estimative Probability’; Wark, ‘The Definitions of Some Estimative Expressions’.

52. Rieber, ‘Communicating Uncertainty … ’.

53. Ben-Israel, ‘Philosophy and methodology of intelligence’; Cooke, Experts in Uncertainty −23.

54. Refsgaard et al., ‘Uncertainty in the environmental modelling process’; Samson et al., ‘A review of different perspectives on uncertainty’.

55. Ben-Israel, ‘Philosophy and methodology … ’.

56. NATO, AJP-2; US DOD, Joint Publication 2–0.

57. Connable, Military Intelligence Fusion; Flynn et al., Fixing Intel; Wentz, Lessons from Kosovo.

58. Rotmans & van Asselt, ‘Uncertainty in Integrated Assessment’.

59. NRC, Review of Recommendations.

60. Schultz et al., Decision making under uncertainty; Stewart, Uncertainty, Judgment, and Error.

61. NRC, Review of Recommendations; Stewart, Uncertainty, Judgment, and Error.

62. Herbert, ‘The Intelligence Analyst as Epistemologist’ 667.

63. Johnston, ‘Analytic Culture’.

64. Betts, ‘Analysis, War, and Decision’; Duvenage, Intelligence Analysis-Knowledge Age; Fingar, Reducing Uncertainty; Friedman & Zeckhauser, ‘Assessing Uncertainty’ and ‘Why Assessing Estimative Accuracy’; Lowenthal, ‘Towards a Reasonable Standard’.

65. Boschetti, ‘A Graphical Representation of Uncertainty’; Skinner et al., ‘A review of uncertainty’; van der Keur et al., ‘Identification of major sources of uncertainty … ’.

66. Friedman & Zeckhauser, ‘Assessing Uncertainty … ’.

67. Sigel et al., ‘Conceptualizing uncertainty … ’; Winkler, ‘Embracing Complexity and Uncertainty … ’.

68. Mangio & Wilkinson, ‘Intelligence Analysis … ’.

69. Borek, ‘Developing an Understanding … ’- 1; ‘Analytic Tradecraft’; ‘Developing a conceptual model … ’.

70. Breakspear, ‘A New Definition’; Clark, Intelligence Analysis; Hulnick, ‘What`s wrong with the Intelligence Cycle’.

71. NATO, AJP-2.1; UK MOD, JDP 2–00.

72. NATO, CCIRM; NATO, AJP-2.

73. DIA, Analytic Tradecraft Guidance.

74. Comprehensively described by e.g., O`Hagan et al., Uncertain Judgments; Cook, Experts in Uncertainty.

75. U.S. EPA, Expert Elicitation- 24.

76. Friedman & Zeckhauser, ‘Analytic Confidence … ’; Peterson, Appropriate Factors.

77. Friedman & Zeckhauser, ‘Analytic Confidence … ’; Irwin & Mandel, ‘Communicating uncertainty in national … ’.

78. Friedman & Zeckhauser, ‘Analytic Confidence … ’.

79. Irwin & Mandel, ‘Improving information evaluation’; Chapter 19.

80. Dhami et al., ‘The analysis of competing hypotheses … ’.

81. Friedman & Zeckhauser, ‘Handling and Mishandling … ’; ‘Analytic Confidence … ’.

82. Isaksen & McNaught, ‘Uncertainty handling … ’.

83. Hanson, ‘The Admiralty Code’; Irwin & Mandel, Chapter 19; Samet, Subjective Interpretation of Reliability.

84. Irwin & Mandel, Chapter 19.

85. Marrin, ‘Understanding and improving intelligence analysis … ’.

86. Borek, Analytic tradecraft … ; Knorr, Foreign Intelligence and the Social Sciences.

87. Bacheș-Torres & Torres-Bacheș, ‘Cross-domain approaches … ’; Marrin, ‘Understanding and improving … ’.

88. Marchio, ‘If the Weatherman Can … ’; Wheaton, ‘Evaluating Intelligence … ’ and ‘The Revolution Begins … ’.

89. Friedman & Zeckhauser, ‘Handling and Mishandling … ’; Javorsek II & Schwitz, «Probing Uncertainty … ”.

90. Lowenthal, ‘Towards a Reasonable Standard … ’-314.

91. e.g. Ben-Israel, ‘Philosophy and methodology … ’; Knorr, Foreign Intelligence … ; NATO, AJP-2.

92. e.g. Dhami et al., ‘Improving Intelligence Analysis … ’; Mandel, ‘Accuracy of Intelligence … ’; ODNI, National Intelligence Strategy.

93. Hill et al., ‘Building understanding … ’-188.

94. Lynham, ‘The General Method … ’; Marrin, ‘Improving Intelligence Studies … ’.

95. Hill et al., ‘Building understanding … ’-187.

96. Bray & Storch, ‘Prediction or Projection?’-535.

97. Dhami et al., ‘Improving Intelligence Analysis … ’; Friedman & Zeckhauser, ‘Why Assessing Estimative Accuracy … ’; Mandel, ‘Can Decision Science Improve … ’.

98. Marchio, ‘How good is your batting average?’; Marrin, ‘Evaluating the Quality … ’.

99. Gentry, ‘The Professionalization of Intelligence Analysis’; Marrin, ‘Understanding and Improving … ’.

100. e.g. Benford et al., ‘Guidance on Uncertainty Analysis … ’; IPCS, Uncertainty and Data Quality.

101. Menkveld, ‘Understanding the complexity of intelligence problems’-4.

102. Coulthart, ‘What’s the problem?’-636–638.

103. Dhami et al., ‘Critical Review of Analytic Techniques … ’.

104. Fishbein & Treverton, Making Sense of Transnational Threats; Moore, Sensemaking.

105. Benford et al., ‘The principles and methods behind EFSA’s Guidance on Uncertainty’; van der Sluijs, ‘RIVM/MNP Guidance for Uncertainty Assessment and Communication’.

106. Marrin, ‘Understanding and improving intelligence analysis…’.

107. Dhami et al., ‘The analysis of competing hypotheses … ’; Mandel, ‘The occasional maverick … ’; Irwin & Mandel, ‘Communicating uncertainty in national … ’.

108. e.g. Marrin, ‘Evaluating CIA’s Analytic Performance’; Phythian, ‘Intelligence analysis today and tomorrow’.

109. Betts, ‘Analysis, War, and Decision’; Davis, ‘The Kent-Kendall debate of 1949’; Gomez, ‘Intelligence reform commissions … ’; Marrin, ‘Rethinking Analytic Politicization’; Scott & Jackson, ‘The Study of Intelligence … ’.

110. Davis, ‘The Challenge of Managing Uncertainty’; Marchio, ‘If the Weatherman can … ’; Phythian, ‘Intelligence analysis today and tomorrow’; Wheaton, ‘Evaluating Intelligence … ’; Wolfberg, ‘When generals consume intelligence … ’.

111. Marrin, ‘Evaluating the Quality of Intelligence … ’.

112. e.g. Doyle et al., ‘Communicating model uncertainty … ’; Etienne et al., Final report: Clear Communications and Uncertainty.

113. Wolfberg, ‘When generals consume intelligence … ’; Lipshitz & Strauss, ‘Coping with Uncertainty … ’; van den Heuvel et al., ‘Coping with uncertainty; Police strategies … ’.

114. Raadgever et al., ‘Uncertainty management strategies … ’.

115. Gabbert et al., ‘Uncertainty analysis in integrated assessment … ’-131.

116. Canton, ‘The Active Management … ’; Immerman, ‘Transforming Analysis … ’.

117. Bruce, Making Analysis … ; Herbert, ‘The Intelligence Analyst … ’; Marrin, ‘Analytic objectivity … ’; Vrist Rønn & Høffding, ‘The Epistemic Status of Intelligence … ’.

118. Marchio, ‘If the Weatherman Can … ’.

119. Marrin, ‘Why strategic intelligence analysis … ’.

120. Wirtz, ‘The Art of the Intelligence Autopsy’.

121. Bacheș-Torres & Torres-Bacheș, ‘Cross-domain approaches’; Vogel & Tyler, ‘Interdisciplinary, cross-sector Collaboration … ’.

122. ODNI, ICD- 205.

123. Heuer & Pherson, Structured Analytic Techniques.

124. Thomson et al., A Typology for Visualizing Uncertainty.

125. Walker et al., ‘Defining Uncertainty: A Conceptual Basis’-6.

126. Boschetti, ‘A Graphical Representation of Uncertainty’-152.

127. Artner et al., Assessing the Value, Chang, ‘Restructuring structured analytic techniques … ’, Coulthart, ‘An evidence-based evaluation … ’.

128. Fisk, The Sino-Soviet Border Dispute; Schweitzer, ‘Bayesian analysis’; Zlotnick, ‘Bayes’ Theorem”.

129. McNaught & Sutovsky, ‘Representing Variable Source Credibility … ’ and ‘Evidence Marshalling … ’.

130. Karvetski et al., ‘Structuring and analyzing … ’.

131. Irwin & Mandel, Chapter 7.

132. Hall & Citrenbaum, Intelligence Analysis −33.

133. US Air Force, Air Force Handbook −29.

134. ODNI, ICD-610- Competency Directories for the Intelligence Community Workforce.

135. Gentry, ‘Has the ODNI Improved … ’ −650.

136. Barnett et al., ‘Analytic rigour in intelligence’- 39.

137. Marrin, ‘Evaluating the Quality … ’.

138. Michealson, MORS Workshop- 7.

Additional information

Notes on contributors

Bjorn G. M. Isaksen

Bjørn G. M. Isaksen is a Faculty Advisor at Norwegian Defence University College, located in Oslo, Norway. Besides being a researcher on intelligence, he also has a military career in which he has been working with intelligence. His PhD addresses core aspects of uncertainty handling within estimative intelligence, and how to better meet contemporary and future quality requirements by developing the extant framework and analytic practices.

Ken R. McNaught

Ken R. McNaught is a Senior Lecturer in Operational Research at Cranfield University’s School of Defence and Security, located at the Defence Academy of the UK. His PhD was concerned with the probabilistic modelling of combat. His research has an applied and inter-disciplinary focus. Current research interests include decision and risk analysis, probabilistic modelling and simulation.