![Sample our Politics & International Relations journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5947/TOC-Subject-Sample-2021-Politics-International-Relations.jpg"})
ABSTRACT
Science and intelligence analysis have a different methodological setting. In science a phenomenon is explained in a general sense, it is in the first place aimed at to explain and to contribute to theory. For that the value of the α is the most critical one: you want to keep the number of incorrect relationships as low as possible. Intelligence analysis is in the first place aimed at not to miss a possible threat. In that research, the value of the β is the most critical one: you want to keep the number of missed relationships as low as possible. Yet, many analytic techniques have been developed in science. These have not been calibrated in order not to miss a relationship. Also reasoning – logic – needs to be reformulated, and calibrated from an α to a β approach. Tooling is needed for a research design into the unknowns.
Disclosure statement
No potential conflict of interest was reported by the author.
Notes
1 A. D De Groot, Methodologie: Grondslagen Van Onderzoek En Denken in De Gedragswetenschappen (Assen: Van Gorcum, 1994), 42, 99.
2 P. J. Van Strien, Praktijk Als Wetenschap: Methodologie Van Het Sociaal-wetenschappelijk Handelen (Assen: Van Gorcum, 1986), 56–58.
3 Van Strien, Praktijk Als Wetenschap, 18–19.
4 Yin, Case Study Research: Design and Methods (Los Angeles: Sage, 2014).
5 Guillaume Gustav de Valk, “Dutch Intelligence – Towards a Qualitative Framework for Analysis” (PhD diss., University of Groningen, 2005) (Den Haag: Boom Juridisch), 66–67.
6 The need to arrange methods and techniques for their β capabilities was first referred to by Giliam de Valk, “Effectiviteit vanuit methodologisch perspectief: welke gevolgen heeft de introductie van nieuwe methoden en technieken?” in Contraterrorisme en ethiek, eds. Michael Kowalski and Martijn Meeder (Amsterdam: Boom, 2011), 69–82.
7 Giliam de Valk, “All-source intelligence,” in Inlichtingen- en veiligheidsdiensten, eds. Beatrice de Graaf, Erwin Muller, and Joop van Reijn (Alphen aan den Rijn: Kluwer, 2010), 530–1.
8 There are handbooks on, for example, Red Teaming. In some handbooks, specialized techniques are also dealt with, for example: Richards J. Heuer and Randolph H. Pherson, Structured Analytic Techniques for Intelligence Analysis (Washington, DC: CQ Press, 2011), §5.7 & §9.6, 122–9, 263–4.
9 Onno Goldbach of the Ministry of Defense and Giliam de Valk at the then Ad de Jonge Centrum, Institute for Interdisciplinary Studies, University of Amsterdam.
10 ”DoD News Briefing – Secretary Rumsfeld and Gen. Myers,” United States Department of Defense, February 12, 2002, https://archive.defense.gov/Transcripts/Transcript.aspx?TranscriptID=2636.
11 Geoffrey K. Pullum, “Language Log,” Language Log: No Foot in Mouth, December 2, 2003, http://itre.cis.upenn.edu/~myl/languagelog/archives/000182.html (accessed January 30, 2012).
12 John Kemp, “Reuters Market Analyst – For Commodities Now,” Commodities Now, March 2011, http://www.commodities-now.com/reports/general/5522-risk-uncertainty-and-black-swans.html (accessed 2012). Kemp refers to Frank Knight – Risk, Uncertainty and Profit, 1921 – and Knightian Uncertainty to explain that economists and insurers have long distinguished between risk and uncertainty. For a matrix representation, see: http://www.ecodigerati.com/content/articles/?page_id=42 (accessed 2012).
13 Robert Ahdieh, “Unknowns Unknowns, Uncertainty, Contracts and Crisis,” in Review of On Uncertainty, Ambiguity, and Contractual Conditions, eds. Eric L. Talley, Jotwell, January 24, 2010, https://corp.jotwell.com/unknown-unknowns-uncertainty-contracts/.
14 Richard C. Walton, “Rumsfield Matrix (Part 1),” http://pdmicrex.blogspot.nl/2010/08/rumsfeld-matrix-part-1.html.
15 An abstract of the explanation of these axes has appeared in: Giliam de Valk, “Case Studies into the Unknown – Logic & Tooling,” Romanian Intelligence Studies Review Vol. . 21 (2019): 243–68. It was presented at an IAFIE conference in 2019 – after the 2016 NISA conference for which this article was presented, and in which the full and elaborate methodology is explained.
16 The composition of a β research design with the Rumsfeld matrix is since 2013 part of the Minor Intelligence Studies, first at the University of Amsterdam [Ad de Jonge Centre] and, since 2017, at the University of Leiden [ISGA].
17 Data: general data, including non-issue specific. Information: issue-specific data. Knowledge: processed and tested information.
18 Generic Early Warning Handbook, Report, EAPC/Council Operations and Exercise Committee (NATO, 2001), 1–97.
19 See, for example, A Military Guide to Terrorism in the Twenty-First Century, Handbook, US Army Training and Doctrine Command, August 15, 2007, https://fas.org/irp/threat/terrorism/guide.pdf.
20 For an explanation, see heading ‘Different classes of reasoning.’
21 Jürgen Simon and Jürgen Taeger, Rasterfahndung Entwicklung, Inhalt Und Grenzen Einer Kriminalpolizeilichen Fahndungsmethode (Baden-Baden: Nomos-Verl.-Ges., 1981). To cope with terrorist groups as the Rote Armee Fraktion, the BKA developed typical unknown-known techniques as Rasterfahndung and Schlebnetzfahndung. It would be interesting to assess if a certain education, company culture, or character structure will lead to a preference to use only certain quadrants of the matrix.
22 Osama Bin Laden’s special operations man, Ali Mohammed, for example, obtained information on unconventional warfare, counterinsurgency operations, and how to command elite soldiers on difficult missions. He was an assigned sergeant with the U.S. Army Special Operations – and unofficially an assistant instructor at the JFK Special Operations Warfare School – at Fort Bragg, North Carolina. He marked some documents as ‘Top Secret for Training otherwise unclassified’ (Steven Emerson, “Osama Bin Laden’s Special Operations Man,” Journal of Counterterrorism & Security International (Fall 1998)).
23 The Red Team Handbook, Report, University of Foreign Military and Cultural Studies, April 2012, https://usacac.army.mil/sites/default/files/documents/ufmcs/The_Red_Team_Handbook.pdf.
24 The British Military devised this technique. In standard patrol flankers, units inspect possible ambush points or dead space areas. For satellite patrol, this is further developed, and intentionally separates itself visually and physically from the base unit of the patrol, outside the visual contact. It demands a better communication and is more difficult to command and control (Urban Operations III: Patrolling. Student Handout. Marine Corps Training Command, https://www.trngcmd.marines.mil/Portals/207/Docs/TBS/B4R5579XQ-DMUrban Operations III – Patrolling.pdf?ver=2016-02-10-114414-840). If terrorists would adapt satellite patrol, they probably would be able to neutralize a series of routine police road blocks.
25 In research, robustness refers to applying several methods and techniques in an analysis. The more such independent tests are performed with a positive outcome, the more plausible the conclusion will be. Consequently, the finding does not depend on the analytical method used. To apply many methods to the same set of data, the margin of error is reduced (Guillaume Gustav de Valk, “Dutch Intelligence – Towards a Qualitative Framework for Analysis: With Case Studies on the Shipping Research Bureau and the National Security Service (BVD),” (PhD diss., University of Groningen, 2005) (Den Haag: Boom Juridisch), 67–68. https://www.rug.nl/research/portal/files/33123437/c9.pdf). It is assumed here that the error margin is not only reduced by applying more methods, but also by applying different classes of methods. There is another reason that calls for robustness of the research design as well. Intelligence deals with events that are sometimes very hard, or even impossible, to predict (‘black swans’). This points to the limitations of knowledge and assessing the future based on the past. It should cause every analyst to question assumptions and test the robustness of its research design. And you should not simply argue ‘after the event that some things were simply unforecastable’ (Kemp, “Reuters Market Analyst”).
26 Two additional remarks need to be made here. First, a well-known danger of inductive logic is illustrated by the observation of the first 100 swans: just because they happen to be white does not mean that all swans are white. Second, mathematical induction, used for mathematical proof, is here arranged as a deductive form of reasoning: in inductive reasoning, the conclusion can be false.
27 Rosa Voulon, Handboek Analyse. Theorievorming En Methodologie in Inlichtingenanalyse (‘t Harde: Defensie Inlichtingen En Veiligheids Instituut, 2009), 24–27. Grabo mentions these three approaches – deduction, induction, and abduction – in relation to the analysis of indications. She puts that it almost always will be a process of abduction (= inference) (Cynthia M. Grabo, Anticipating Surprise: Analysis for Strategic Warning (Lanham: University Press of America, 2004), 42–43).
28 De Groot, Methodologie, 38, 76–82, 38; Grabo, Anticipating Surprise, 42–44; and Voulon, Handboek Analyse, 24–27.
29 In such experiments, the unknown-unknown–is addressed, by speaking of a residual threat. Practitioners use so-called Red Team and Red Cell experiments to reduce the residual threats. It deviates from scientific experiments in which a hypothesis is tested – and, by that, is related to the α.
30 Giliam de Valk and Willemijn Aerdts, “Inlichtingenwerk Vanuit Een Methodologisch Perspectief,” Justitiële Verkenningen 44, no. 1 (2018): 120–122.
31 Giliam de Valk, “Red Team and Science” (Presentation for De Nederlandsche Bank (DNB), Den Haag, June 8, 2012).
32 Richard Heuer, “Biases in Evaluation of Evidence,” Studies in Intelligence (Winter 1981) Box 8, 92-3: 31–35.
33 ‘This process is as follows: if you receive evidence, you postulate a set of causal connections that explains this evidence. The stronger you perceive the relation between facts leading to that causal connection; the stronger you perceive that causal connection. This attribution tends to persist even after the evidence that created those connections has been fully discredited. Even if you learn that the information – on which you developed your causal connection – comes from an uncontrolled source who may be trying to manipulate you, this does not necessarily reduce the impact of this causal connection. In general, the ‘early but incorrect impression tends to persist because the amount of information necessary to invalidate a hypothesis is considerably greater than the amount of information required to make an initial interpretation […] People form impressions on the basis of very little information, but once formed, they do not reject or change them unless they obtain rather solid evidence’ From: de Valk, Dutch Intelligence, 82. De Valk is referring here to Heuer, ‘Biases in Evaluation of Evidence.”
34 To derive conclusions from possible hypotheses, and to invert conditionals to apply probabilistic abduction.
35 For the mentioned techniques, see: Heuer and Pherson, Structured Analytic Techniques, 140–3, 163–6.
36 Arthur Hulnick, “The Intelligence-Producer-Policy Consumer Linkage,” Studies in Intelligence (Winter 1985) Box 9, 108-7: 76–9.
37 See, the .
38 You will try to get your data bases filled in an optimal way by starting to monitor those aspects or places that will yield the most relevant information. For this, GEOINT or a theoretical steered approach such as the choke point theory [this is a so-called B-theory] can be made use of (Giliam de Valk, “All-source Intelligence,” 507–33, 527–8).
39 As put, this is needed, among others, for the persistence of impressions based on discredited evidence in the causal connection (Heuer, “Biases in Evaluation of Evidence,” 44–46).
40 For an explanation, see further down this section.
41 Nuclear Security Summits (NSS) are originally an initiative of President Obama. In 2010 he invited state and government leaders of a large number of countries in order to cope the threat of nuclear terrorism, and to fight illegal trade in nuclear material “Uw Verzoek Inzake Top Nucleaire Veiligheid 2014 (NSS 2014/2019),” Frans Timmermans to Tweede Kamer, April 26, 2013.
42 SLEIPNIR is ‘an analytical technique developed to rank order organized groups of criminals in terms of their relative capabilities, limitations and vulnerabilities. The rank ordered lists of groups are components of strategic intelligence assessments used to recommend intelligence and enforcement priorities’. To cope with organized crime, for example, attributes are selected as (in rank): 1 corruption; 2 violence; 3 infiltration; 4 expertise; 5 sophistication; 6 subversion; 7 strategy; 8 discipline; 9 insulation; 10 intelligence use; 11 multiple enterprises; 12 mobility; 13 stability; 14 scope; 15 monopoly; 16 group cohesiveness; 17 continuity; 18 links to other organized crime groups; 19 links to criminal extremist groups. For terrorism, a different attribute set is made (Steven J. Strang, Project SLEIPNIR: An Analytical Technique for Operational Priority Setting (Ottawa: RCMP, 2019), 1–5).
43 Because of the size of such a summit, different locations to meet will be used, as will participants dine and sleep at different locations. Not only these locations can be targeted, but also the supply and transportation lines of a summit. As opponents can choose between multiple targets, an analysis is needed of the target selection from the perpetrator’s perspective.
44 CARVER + Shock is an acronym for criticality (measure of impact of an attack), accessibility (ability to physically access and egress from target), recuperability (ability of system to recover from an attack), vulnerability (ease of accomplishing attack), effect (amount of direct loss from an attack as measured by loss of production) and recognisability (ease of identifying target) + shock (the combined health, economic, and psychological impacts of an attack). It is a prioritization tool, a system of target acquisition, to rank potential targets according to a scale. By identifying and ranking the potential targets, attack resources can be efficiently used. It assesses the vulnerabilities within a system, industry, or infrastructure. Originally developed for US special forces, it is now also applied by, among others, the US Food and Drug Administration to enhance ‘food defence’ Consumer Updates, accessed 2012, http://www.fda.gov/ForConsumers/ConsumerUpdates/ucm094560.htm; and accessed 2012, http://www.fda.gov/Food/NewsEvents/ConstituentUpdates/ucm180608.htm.
45 In Red Team/Red Cell exercises the University of Amsterdam carried out, infosec and opsec was a returning point of attention. Sometimes, technical personnel – crucial for the maintenance of the infrastructure – complained on social media about their managers. In other cases, the target organisation had opsec and infosec intact, but it others, like local municipalities and provinces, published sensitive material on their sites that could be used to plan a terrorist attack.
46 In 1962, H. A. Watson of Bell Telephone Laboratories developed the Fault Tree Analysis – also referred at as Event Tree Analysis – for the US Air Force (Minutemen). It is a logic diagram to relate conditions that precede faults and undesired events. At the top of the schedule, the undesired event – end state – is placed. It can be applied in both a qualitative and quantitative way: Anna L. Martensen and Ricky W. Butler, The Fault-Tree Compiler (Hampton, VA: National Aeronautics and Space Administration, Langley Research Center, 1987), 1–3, 6–9.
47 The Quantitative Intrusion Path Analysis is a method that is known under different synonyms – with many different variants – often referring to the name of the specific software that is used to carry it out. It is designed not only to weigh physical security measures, but also the human factor. Thus, it could be measured if an opponent could enter – and at what speed, by what AMO – secured critical infrastructure. It measures the delay by physical barriers, and calculates issues as recognition, warning and reaction time (for a full system case study, see, for example: PR&PP Evaluation: ESFR Full System Case Study Final Report, Report, Proliferation Resistance and Physical Protection Evaluation Methodology Working Group, October 2009, https://www.gen-4.org/gif/upload/docs/application/pdf/2013-09/prpp_csreport_and_appendices_2009_10-29.pdf). It is also worked out in several variations for cyber.
48 A Quantitative Intrusion Path Analysis (QIPA) would also have been helpful in case of, for example, the art robbery at the Rotterdam Kunsthal. In October 2012, a gang robbed paintings – and later partly burned these – by Claude Monet, Pablo Picasso, Henri Matisse, Paul Gauguin, Meyer de Haan and Lucian Freud, worth many millions of euros. Although there was, rightly so, a lot of criticism on the poor state of the locks on the doors, the robbery was an analytic failure in the first place. There was an alarm, but you should have calculated the delay that each security ring would have confronted the robbers with. Then you combine it with the time the reinforcement (police/security) needs to be in place. Only by such a combined analysis, a sound security plan can be designed. And QIPA is per excellence a technique to calculate that.
49 In the intelligence literature, some publications deal in-depth with the issue of the formulation of the research question, for example: Heuer and Pherson, Structured Analytic Techniques, § 4.3; and Brian Manning and Kristan Wheaton, “Making “Easy Questions” Easy: The Difficulty of Intelligence Requirements,” International Journal of Intelligence and Counterintelligence 26, no. 3 (September 2013): 597–611.
50 The logic of this freedom of formulating the research question lies in the type of quadrant – the unknown-unknown. In practice, at the Ad de Jonge Centre such limitations were felt as a result of the scope set by the authorities.
Additional information
Notes on contributors
Giliam de Valk
In 2005, Giliam de Valk published his PhD on the quality intelligence analyses have to meet. He is specialized in the methodology of security and intelligence analysis. He has worked at the University of Amsterdam, the University of Utrecht, and the Netherlands Defense Academy where he coordinated and lectured a minor on intelligence studies. At the moment he is an assistant professor at the Institute for Security and Global Affairs, Leiden, Leiden University
Onno Goldbach
Onno Goldbach After finishing his Master’s degree in Physical Geography, Onno Goldbach joined the Royal Netherlands army as a geospatial analyst. During his service at the Dutch Defense Intelligence & Security Institute, Harde, he met academic counterparts. One of them was Giliam de Valk, with whom he surveyed and worked out innovative ideas. This article is one of these ideas.