Weighing the evidence: the BCISS Iraq HUMINT Analytic Matrix Exercise

Abstract

This article examines the Brunel Iraq HUMINT Matrix exercise. The purpose of this approach to intelligence pedagogy is to get participants to think through and work out analytic methods, issues, and potential solutions from first principles and for themselves. Our strategy is to try and fuse training and education learning outcomes, so that students emerge with a technical competence in analytic methods, underpinned by a deeper understanding of the foundations and internal logic shaping those methods. The Iraq Matrix exercise seeks to unpack and examine the nuts and bolts of source evaluation, and to test alternative hypotheses with particular attention to the relationship between the quality of various sources and, the weight of judgements they can or cannot sustain. The ultimate goal is to encourage what is currently fashionably referred to as ‘reflexive practice’, whereby the practitioner reflects critically and self-critically upon how their task works and how they do it, then uses those insights to improve their workplace performance. But not all of our teaching is directed towards practitioners. For those whose aims are scholarly and academic, the aim is to give observers a more visceral understanding of the challenges of the intelligence task they intended to study. Here the intended reflexive practice goal is to encourage an empathy with the workaday challenges facing those in the business of intelligence analysis, and to discourage the observer’s temptation to make facile and simplistic judgements about processes or events.

Notes

1. Mencken, Prejudices: Second Series, 120.

2. Davies, “Assessment BASE: Simulating National Intelligence Assessment,” 721–36; and Davies, “Jointery versus Tradecraft,” 203–22.

3. For an independent account, see Palacios, “Intelligence Analysis Training,” 48.

4. Marrin, Improving Intelligence Analysis, 77.

5. (London: The Stationery Office (TSO), 2004), hereafter just referred to as the Review.

6. This is a recurrent theme throughout report of the Presidential Commission on the Intelligence Capabilities of the United States, Regarding Weapons of Mass Destruction (aka WMD Commission) Report to the President of the United States but is particularly explicit on 169.

7. Palo Alto Research Centre. ‘ACH_2.0.5 Download Page’ http://www2.parc.com/istl/projects/ach/ach.html (accessed 14 April 2017).

8. Since acquired by IBM as i2 Analyze, see https://www.ibm.com/us-en/marketplace/enterprise-intelligence-analysis (accessed 17 April 2017).

9. https://www.palantir.com/ (accessed 17 April 2017).

10. https://www.paterva.com/web7/ (accessed 17 April 2017).

11. For which the field has been sternly criticised by, e.g. Erik Dahl in his Intelligence and Surprise Attack.

12. Davies, “Collection and Analysis on Iraq,” 41–54.

13. US Government, A Tradecraft Primer, 14, 15.

14. For an overview of Strong’s background and immediate post-war intelligence career see Dylan, “Thinking About Defence Intelligence,” 836–43.

15. Strong, Men of Intelligence, 40. The authors are indebted to Matt Jolly and to a former senior official at DIAS for bringing Strong’s account to our attention.

16. See, e.g. Clausewitz, On War, 117.

17. Jomini, Art of War, 203.

18. Jomini, Art of War, 201.

19. Strong, Men of Intelligence, 153.

20. Strong, Men of Intelligence, 40 infra.

21. Heuer and Pherson, Structured Analytic Techniques, 140–3.

22. A point Heuer and Pherson themselves note Structured Analytic Techniques, 141.

23. See, e.g. MacEachin, “The Tradecraft of Analysis,” 63–74.

24. See, e.g. United States Army, FM 34–60 Counterintelligence, 45–8 or more recently Joint Chiefs of Staff, JP 2–01 Joint and National Intelligence, D9–D13.

25. Stober and Hoffman, A Convenient Spy, 235; significantly this method failed in the investigation of Chinese acquisition of details concerning the W88 thermonuclear warhead largely because of the large number of individuals with access to information on the programme.

26. Booth, State Department Counterintelligence.

27. For example, Heuer, The Psychology of Intelligence Analysis, 19.

28. House of Commons Foreign Affairs Select Committee, The Decision to Go to War in IraqHC 813-I, 48–50.

29. Intelligence and Security Committee, Iraqi Weapons of Mass Destruction: Intelligence and Assessments; on partisan political influence especially on the then-Chair, Anne Tayler MP see e.g. Glees and Davies, “Intelligence, Iraq and the Limits of Legislative Accountability,” 848–83.

30. Hutton, Report of the Inquiry into the Circumstances Surrounding.

31. Butler, Review.

32. ‘Whitehall Mandarin’ is a colloquialism of the British political classes denoting a certain kind of cautiously phrased, clinical and sometimes technocratically opaque style of drafting considered typical of the Home Civil Service.

33. Fortunately also available as a pdf on an open license; Chilcot, Report of the Iraq Inquiry, downloadable pdf at: http://www.iraqinquiry.org.uk/the-report/ (accessed 17 April 2017).

34. Considerably more detail was, however, given to the finished JIC assessments.

35. As Butler notes at a number of points the Review, the lion’s share of UK assessments on Iraq WMD dealt with and pointed towards a ‘break out capability’ more than anything else pp.41, 48, 63, 75, 116, 130, 157 and passim.

36. This appears to have resulted from last minute edits trying to present SIS sources in a manner that sufficiently protected their identity. Private information.

37. Butler, Review, 100, 101, and 107, 108 respectively.

38. Note that the order in which these sources are treated follows that in Butler’s, Review and not that used in Davies, “Collection and Analysis on Iraq,” 46–58.

39. Butler, Review, 100, 107.

40. Butler, Review, 75.

41. Butler, Review, 100, 107.

42. Butler, Review, 100.

43. Butler, Review, 107.

44. Private information.

45. Butler, Review, 61.

46. Butler, Review, 100, 102, 108.

47. Another confusing feature of the Butler report is the appearance of sources described as shown to be reliable in 2004 appearing as ‘new’ sources at the times of their recruitment two or three years previously.

48. In the phasing used in a JIC assessment of 10 May 2001, Butler, Review, 56.

49. Butler, Review, 60.

50. Butler, Review, 60 infra.

51. Butler, Review, 100, 101.

52. Butler, Review, 101.

53. Woodward, Plan of Attack, 219.

54. Referred to in the CIA Primer as a ‘quality of information check’ pp.10–11.

55. Butler, Review, 61.

56. Butler, Review, 56.

57. Much more commonly seen is the older ‘5 × 5’ grading convention.

58. Development, Concepts and Doctrine Centre (DCDC), JDP 2-00: Understanding and Intelligence Support.

59. See, e.g. Davies, MI6 and the Machinery of Spying, 287, 338, 339.

60. Professional Head of Defence Intelligence Analysis (PODIA), Quick Wins for Busy Analysts, 15.

61. PODIA, Quick Wins, 17.

62. In most implementations of this practical by BCISS the matrix exercise is preceded by at least one lecture on the fundamentals of social science research concepts and methodology.

63. Steele, “The Trump Russia Dossier”. For the original publication, see Bensinger et al., “These Reports Allege Trump Has Deep Ties To Russia.”

64. Henderson et al., “Donald Trump Attacks.”

65. Tracy, “What Intelligence Experts Think of the Explosive Trump-Russia Report.”