Towards a robust β research design: on reasoning and different classes of unknowns

ABSTRACT

Science and intelligence analysis have a different methodological setting. In science a phenomenon is explained in a general sense, it is in the first place aimed at to explain and to contribute to theory. For that the value of the α is the most critical one: you want to keep the number of incorrect relationships as low as possible. Intelligence analysis is in the first place aimed at not to miss a possible threat. In that research, the value of the β is the most critical one: you want to keep the number of missed relationships as low as possible. Yet, many analytic techniques have been developed in science. These have not been calibrated in order not to miss a relationship. Also reasoning – logic – needs to be reformulated, and calibrated from an α to a β approach. Tooling is needed for a research design into the unknowns.

KEYWORDS:

• Intelligence analysis
• methodology
• logic and reasoning
• β
• unknowns
• tooling
• Rumsfeld matrix
• case research

Introduction

At the NISA conference, the central focus was on the analyst and analysis. Intelligence analysis differs – from a methodological point of view – significantly from science, and also from many other disciplines. Scientific and intelligence research differ in nature concerning both theory and methodology. This article deals with these differences and its consequences for a research design for intelligence analysis.

Theory

In science, academics usually understand theory as a general or nomothetic theory. A phenomenon is explained in a general sense. This type of theory is referred to as a level-A theory.¹ In applied research, professionals also develop a theory, but in the form of a level-B and level-C theory. The level-B theory is a problem oriented special theory, and its explanation of a phenomenon is limited to a certain category of cases. The level-C theory is developed for an individual case. This is also referred to as an N = 1 theory – or an idionom theory.²

The level-C theory is likely to be used by intelligence analysts analysing a concrete case. Intelligence research aims at actions concerning future situations, and not at scientific theory. Intelligence analysis is primarily aimed at interventions, contrary to scientific analysis that is primarily aimed at truth finding. Intelligence analysis aims to realize a situation that is believed to be the desired one. It focuses on factors that can be manipulated, and by that the object of research is more mutandum than explanandum.³

But how do we compose a C-theory in intelligence analysis? There are high quality publications on case-study research, as by Robert K. Yin,⁴ but they are not calibrated to the specific methodological needs of intelligence research. His publication is written for scientific purposes in order to explain. Intelligence is, however, in the first place aimed at not to miss developments. To explore this, some methodological aspects of intelligence research need to be dealt with first.

The α and β

Intelligence research is carried out in a theatre in which the opponent tries to confuse the analyst by denial and deception. Intelligence analysis is in the first place aimed at not to miss a possible threat. The aim of the analyst is to give warnings to avert a threat.

These differences in theatre and orientation has consequences for how such a research deals with the so-called α and β. The α is the chance that you incorrectly conclude that there is a significant relationship between phenomena. The β is the chance that you do not discover a weak, but actual existing, relationship between phenomena. In traditional academic research, the emphasis is to reduce the α – the chance that you incorrectly conclude that there is a significant relationship between phenomena. In intelligence research, however, the main emphasis is on not to miss a threat – the β – the chance that you do not discover a weak, but actual existing, relationship between phenomena.⁵

To put it in plain language: it is often more critical that you do not miss a threat [β orientated research], than that you scientifically prove or explain that a threat will occur [α orientated research]. This calls for a research design, and the application of logic, methods and techniques, with respect to β capabilities.

β gap

There has been a revolution of introducing all kinds of methods and techniques from science in intelligence analysis. These academic methods and techniques were mostly developed for their α capabilities, rather than for their β capabilities.⁶ This difference in orientation has three consequences.

First, in intelligence analysis the value of the α is generally high, the analyst does not need to prove his case as in-depth as in science before taking action. So, it is relatively easy to integrate techniques designed in science – with a low value of the α – into the analysis.

Secondly, the value of the β in intelligence is much more critical than in science, as the intelligence analyst does not want to miss relationships that point to a threat. If the analyst uses a technique that is developed in science, there is a possibility that it is not calibrated to the more critical value of the field of intelligence. In that case, the intelligence analyst will miss more than in the case for a calibrated technique. And, it is possible, the analyst is not even aware of this situation.

The combination of the first two points, may lead to a third consequence. The analyst may get an incorrect feeling of accuracy. In that case, the analyst is, by the application of new uncalibrated techniques, more able to increase the robustness of the research. As a result, the analyst will sooner reach the desired value of the α. It gives the analyst the feeling to have a better grip on the situation, while at the same time threats will be missed by the high value of the β in scientific techniques.⁷ It leaves us with the puzzle of how to compose a β research design, in order not to miss a threat. Or, how do we deal in intelligence analysis with the β gap caused by the introduction of scientific methodology?

To compose a β research design, we will have to deal with logic and unknowns. First, there is logic, and the consequences of applying logic if you focus on reducing the β. Secondly, there are the unknowns, and how to deal with them. What we wonder is, whether a β research design is possible.

There are no manuals on a β research design. Some publications concern β-aspects, for example when they deal with techniques as Quadrant Crunching, Red Team, Red Cell or Alternative Analysis.⁸ However, the β research design itself is a blind spot. In the Netherlands, some initiatives were taken to explore the possibilities of such a β research design.⁹ Starting point was to distinguish between different types of unknowns – whether the way to retrieve data is known or not, and whether these data themselves are known or not.

The Rumsfeld matrix: the unknowns

To find out more about a β research design, a quote by Donald Rumsfeld is the starting point. In 2002, the then United States Secretary of Defense, Rumsfeld stated:

[T]here are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – there are things we do not know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.¹⁰

Rumsfeld’s quote gave rise to a public debate, including an academic one. Rumsfeld made his remarks in the context of the Iraq war and the absence of evidence of weapons of mass destruction. One of the reactions was by Geoffrey K. Pullum, professor of General Linguistics at the University of Edinburgh. Pullum posed that Rumsfeld’s quote was ‘impeccable, syntactically, semantically, logically, and rhetorically’.¹¹ The emphasis on problems caused by unknowns can make the Rumsfeld statement interesting to explore for its potential to arrange methods and techniques for their β capabilities. A Rumsfeld matrix derived from what he said that day, is used more often, e.g. in order to explain the distinction between risk and uncertainty;¹² the awareness of unknown unknowns to function as an independent determinant of our utility;¹³ and, e.g. in management; in order to rethink the difference between risk and observability.¹⁴ However, the Rumsfeld matrix has never been used for a methodological arrangement of β capabilities yet.

Goldbach & De Valk transformed the quote of Rumsfeld into two axes of unknowns.¹⁵ The x-axis shows whether the data are known or not [data]. The y-axis shows whether the way to retrieve data is known or not [retrieval].¹⁶ It leads to four combinations of retrieval [known/unknown] and data [known/unknown] (See Table 1). Each of those combination is a quadrant of the matrix. If his statement is thus rearranged in a matrix, it results in the following composing elements:

Table 1. The Rumsfeld matrix: data and retrieval

Data Retrieval	Known	Unknown
Known	Known-Known	Known-Unknown
Unknown	Unknown-Known	Unknown-Unknown

The only field that is not covered by the original Rumsfeld statement itself is the unknown-known – things that you are not aware of that you know them. However, from the perspective of logic, there is no reason to exclude this option. It will therefore be part of the analysis.

The aim of this exercise is to explore whether or not the value of the β can be reduced by developing a new tool. This instrument will distinguish different types of unknowns. Each type of unknowns refers to a certain combination of elements that you miss, which may pose a risk or a threat. Each quadrant covers a part of the puzzle of the [un-]known-[un-]known, and is part of the research to reduce the value of the β as much as possible.

Known-Known

In the Known-Known quadrant, both the technique to obtain the data [retrieval] and the data are known. The known-known quadrant refers to the contents of data, information, and knowledge¹⁷ of which we know that we know them. It also encompasses assumptions we work with, or relationships which we think are established ones.

A known-known can be perceived as something absolutely certain, as ‘the sun will rise every morning,’ or something probabilistic, as ‘a dice has a 1/6 chance to land on the 1.’ Conceptually there is no difference. As a dice has a 1/6 chance to land on the 1, it belongs to the known-known quadrant, because the claim is to be certain about the probability. If you are playing in an illegal casino, the dice may be prepared. That such a dice can be tricked is then the challenge that is put in the known-known quadrant.

In general, this quadrant is about challenging: are you really sure about the things you think that you know that you know them? If the known-knowns are really known-knowns, such contents have to be evaluated. This not only applies to the data themselves [reliability and credibility], but also to critical thinking concerning assumptions, established patterns, and so on. Within Indicator & Warning systems, for example, you need to check if the so-called Critical Indicators of a Warning Scenario are still accurate. The accuracy of a Critical Indicator is vital in order that a scenario will provide a warning in case of threats.¹⁸ Critical Indicators in the context of a specific Warning Scenario are related to the α – as they assess the probability of that specific Warning Scenario. This is contrary to the so-called Suspicious Indicators that indicate if a threat could possibly occur. For more on the Suspicious Indicator, see Known-Unknown.

Known-Unknown

In the Known-Unknown quadrant, the technique of retrieval is known, but the data themselves are still unknown. To assess them, often a top-down inventory can be made to identify conditions and factors that can contribute to or result in a threat. Within such an inventory, it is unknown beforehand what it exactly may go wrong, or how large the mayhem will be. This will depend on factors such as an adversary modus operandi [AMO]. To put it in more general terms: you are aware the threat is present, but the exact contents, character, modus operandi, approach, and aims are unknown, uncertain, or unspecified. In order not to miss such threats, methods and techniques are used to assess and to cope with the enemy’s course of action [ECOA].

In this quadrant techniques are used, for example, to identify passengers at airports who want to carry out attacks. In such an analysis, regularly, it is attempted to recognize AMO’s through suspicious indicators. Often, this is carried out by a combination of Predictive Profiling and Security Questioning. For Predictive Profiling Suspicious Indicators are developed with the help of the Terrorist or Criminal Planning Cycle.¹⁹ In this cycle, the AMO’s are identified for each of the different steps of preparation and execution. At every step, suspicious indicators are developed for each of the AMO’s, in order to unveil a possible activity in an phase as early as possible. After identification, an intervention may take place. Often, this intervention is a so-called Security Questioning – in which the security officer tries to deny that the passenger wants to carry out an attack, or something similar.

Unknown-Known

In the Unknown-Known quadrant, the technique or algorithm to obtain information [retrieval] is unknown, but the data as such are present. In this case, the issue is about finding relevant correlations. In these cases, not knowing these is due to the absence of the right tooling or exploitation of these databases. Identifying these can be carried out through so-called big data-analyses. A variation of such an imperfect exploitation is the absence of connecting databases, by which either information is simply not connected, or the set of data is not large enough to get significant correlations. Often, such unknown-knowns can be considered as organisational intelligence gaps because it is caused by either unconsulted resources such as databases, available literature, or unrecognised subject matter experts, or data available in compartments that cannot be shared due to intelligence laws, competition, egos, etc. In intelligence, tooling is developed for, among others, data mining, criminal profiling, geographic profiling, spatial analysis, social network analysis, and GEOINT.

In practice, algorithms are applied at large data-bases to map possible relevant correlations. Abduction – or inference to the best explanation²⁰ – is an important way of reasoning in this quadrant. This quadrant is not included in Rumsfeld statement. However, the German Bundeskriminalamt [BKA] applied it already in the 70’s when it was looking for, for example, the refuges of the then Rote Armee Fraktion.²¹ The added value of this quadrant is that large numbers of correlations can be identified that otherwise would have been overlooked by the analyst. This generation of correlations can be carried out on a scale that is hardly possible in the other quadrants. By that, this so-called data-mining is a crucial element to reduce the β – and by that to reduce the chance that you overlook threats.

Unknown-Unknown

In the Unknown-Unknown quadrant, both the technique to obtain information [retrieval] and the data are unknown. This category tends to be the most difficult one. Not only for the obvious reason that it is hard to reflect on things you do not know of, but also because there are hardly any techniques developed to trace unknown-unknowns.

Tracing them mainly takes the form of an experiment. In such experiments a group of persons is asked to carry out an authorized attack on their own organization, to see if something is overlooked in the defence or security measures. Such an experiment is often referred to as Red Team or Red Cell. The main aim in those experiments is to figure out if new AMOs can be discovered. Such an AMO may already have been applied in another ‘discipline’ than by, for example, a terrorist group, such as the military. Especially military manuals are popular in terrorist groups, in which they look for new AMO’s.²² Red Team/Cell, however, is not limited to the opponents perspective, but can also include the broader scope of society itself, including secondary and tertiary effects.²³

Induction is an important way of reasoning of this quadrant. Based on the results of a Red Cell experiment, general security measures are taken. For example, the outcome of a Red Cell experiment can be that terrorists may adapt satellite patrol²⁴ to neutralize police road blocks. Subsequently, these road blocks are organized in a different way, to cope with this new AMO.

After explaining the different types of unknowns, it will now be assessed how reasoning can be put in the context of reducing the β.

The Rumsfeld matrix: reasoning

To get a better feeling of how a β research design is carried out, an impression is given of how an ideal Rumsfeld matrix is composed and what this means for the results of any investigation based on it.

For an optimal coverage of a given research question, it is assumed – for reasons of robustness²⁵ – that all four the different quadrants of the matrix are filled in, and, in doing so, all different classes of reasoning have to be used. These classes – deduction, induction, and the inference to the best explanation [IBE] – are explained under the next heading. If such an arrangement of different classes of reasoning is possible, it will not result in more of the same type of reasoning, but in a real robust – β orientated – research approach. As every class of reasoning has biases or limitations, these are likely minimized by combining different classes of reasoning in a research rather than using more techniques of the same class, since these are likely to have the same bias. Moreover, by using different classes of reasoning, it is more likely to cover different aspects of the research question. In addition it is recommended to combine both quantitative and qualitative elements – if possible.

Different classes of reasoning

Having made these general remarks on the classes of reasoning, a short description is presented. First, there is deductive reasoning. In deductive reasoning you argue from the general to the specific – a top-down approach. In a logic way, the conclusions are deductive of the premises presented. An argumentation is deductive, meaning that if the premises are correct, the conclusion therefore will inevitably also be correct. Secondly, there is inductive reasoning. Here, a general rule – generalization – is made based upon a number of specific observations, experiments, etc. These observations and experiments indicate that the premises of an inductive logical argument have some degree of support. It is a bottom-up approach. It is also referred at as the ex-consequentia reasoning. The conclusions that result from inductive reasoning – and in which the premises are true – can be true, but also false. This can be explained as follows. If the premises are true the conclusion is likely to be true. Inductive reasoning is probabilistic, the premises do not make the conclusion absolute.²⁶ Thirdly, there is the inference to the best explanation, or abductive reasoning, in which an explanation is selected based upon likeliness. In abductive reasoning, it is assumed that the most likely conclusion is the correct one.²⁷ It is reasoning through successive approximation.

Table 2. Reasoning and reducing the value of the β

Reasoning	Strength	Weakness
Deduction	Fast, first, general inventory on already identified threats. It directs the research at possible residual threats. Good at calculating the resilience.	Weak in making an inventory of a deviation of the general pattern. Hardly covers real innovations.
Induction	Maps innovations, as new AMO’s. Aims at the unique [e.g. by Verstehen].	Slow for making an inventory of possible threats. Maps only a small part of the case [= C-theory].
IBE [abduction]	Big data [quantitative]: generates many correlations, otherwise overlooked. These correlations may lead to additional hypotheses and maps patterns with a prognostic value [trends].	Often no causality. A minority of the correlations will be of relevance for the threat issue. Often, additional qualitative checks are needed. Limited mapping of innovations [only for trends], because the data need to be present in a significant number.

Table 3. The Rumsfeld matrix: classes of reasoning

Data To retrieve	Known	Unknown
Known	IBE [Abductive Reasoning] (as in belief revision [: to take into account a new piece of information])	Deductive Reasoning (as in Fault Tree Analysis) IBE [Abductive Reasoning] (as in Analysis of Competing Hypotheses, Indicators Validator, Bayesian Networks)
Unknown	IBE [Abductive Reasoning] (as in criminal and geographic profiling) Inductive Reasoning (as in statistical syllogism)^1	Inductive Reasoning (Red Team/Cell)

¹A syllogism (Greek for conclusion or inference) is a logical argument in which a conclusion is based on two or more propositions that are asserted or assumed to be true. Unlike many other forms of syllogism, a statistical syllogism is inductive, so when evaluating this kind of argument it is important to consider how strong or weak it is, along with the other rules of induction (as opposed to deduction)

As formulated above, α research is orientated towards the process of how to reach your conclusions and on the absoluteness of your claim. β research, however, is not about explaining and proving, but on not to miss relationships. In such a context, methods – used in a deductive, inductive way or via inference to the best explanation – need to be reformulated, and calibrated from an α approach to a β approach. This mainly has to do with how to reach your claim – it is not about wanting to proof or to explain, but about not to miss a relationship. Not only in general publications on methodology, but even in intelligence handbooks, reasoning is only presented and explained in the context of the α,²⁸ and not the β. And, you would hardly find anything methodological on how to design a research that is primarily aimed at reducing the β.

To what extent can reasoning – deduction, induction, and inference to the best explanation – contribute in order not to miss relevant relationships: in case of e.g. threats? At the Ad de Jonge Centre, University of Amsterdam, during Red Team and Red Cell experiments,²⁹ some insights were obtained how reasoning may contribute to reduce the chance we will miss a threat – i.e. to reduce the value of the β. To what extent will contribute reasoning – from the general to the specific [deductive], reasoning from the specific to the general [inductive], and inference to the best explanation – to not to miss relevant relationships? Without claiming definitive conclusions, the experiments at the Ad de Jonge Centre indicated some strong and weak points of these three ways of reasoning (See Table 2).³⁰

Concerning the reduction of the value of the β, the preliminary findings at the Ad de Jonge Centre seem to point at unique weak and strong points for each type of reasoning. It supports the earlier general statement that it is advisable always to use all three forms of reasoning in order to reduce the β.

Reasoning and the Rumsfeld matrix

If we put the different classes of reasoning in the Rumsfeld matrix, certain classes seem to occur more often in some quadrants than in others. Deductive reasoning often takes place in the known-unknown quadrant, whereas it is rare in the unknown-known one. The unknown-unknown quadrant seems almost exclusively to rely on inductive reasoning. This may not come as a surprise, since Red Team/Red Cell can be described as a β experiment, to identify threats that belong to the category of the unknown-unknown quadrant, to assess if that threat is possible or not: 0 or 1.³¹ If we would compose a matrix in which the classes of reasoning are put that are dominant in a certain quadrant, it would look as follows (See Table 3).

Some additional remarks with regard to this this table need to be made. First, other classes of reasoning are not completely excluded, but it deals with the general preference of a certain quadrant. Second, to answer a research question, does not require to limit yourself to one technique in a quadrant. As deductive reasoning more often occurs in the known-unknown quadrant, it is recommended to apply at least one technique of this class in this quadrant. Third, it is recommended to fill in each quadrant with at least one technique, and to use all three classes of reasoning to answer your research question. Finally, it may be pointed out that this table is far from complete with concern to methods and techniques. A few examples will be presented later on in this article.

In addition, with regard to inductive reasoning, there is a tendency that you rely on information that is vivid, concrete, or personal information. Case studies and anecdotes tend to have a greater impact on intelligence analysts than statistical information which actually have greater evidential value.³² By doing so, there is a risk of a biased outcome and overlooking alternatives and other existing relationships. This may lead to an intelligence failure. To cope with such potential pitfalls, it is recommended to add in the matrix techniques that challenge assumptions – such as the role of Devil’s Advocate or belief revision as part of the known-known quadrant techniques.

Techniques that challenge assumptions can be found, for example, in the class of abductive reasoning. Abductive reasoning is applied for belief revision – and also an update of a certain belief – in which opinions are challenged and changed as a result of new incoming data that are inconsistent with the existing views. Belief revision is therefore a useful technique to cope with a typical intelligence bias, like the persistence of impressions based on discredited evidence in the causal connection.³³ Abductive reasoning has sub-groups, such as probabilistic abduction,³⁴ and subjective logic abduction in which the uncertainty of the input elements is also included. In intelligence analysis, abductive reasoning is used, for example, in Bayesian Networks and in Analysis of Competing Hypotheses. Elements of subjective logic abduction can be traced in some applications of Analysis of Competing Hypotheses, or in the Indicators Validator.³⁵

To summarize, an optimal Rumsfeld matrix is not composed of more of the same type of methods, but it is filled in with variety of methods from different classes of reasoning as wide as possible. Through such a robust approach, an arrangement with the help of the Rumsfeld matrix can live up to its expectations to reduce the number of relationships that otherwise would have been overlooked.

Sequence of research

To answer a given research question, the Rumsfeld matrix is approached as a matrix, and not as a cycle, in the sense that techniques are not applied in a sequential, but in a parallel way. In this, there is a resemblance to Arthur Hulnick’s intelligence matrix that – like the Rumsfeld matrix – is product-oriented. This is contrary to the more often quoted intelligence cycle that is a phase model which is more of an abstract interpretation, than an actual representation.³⁶

There are two exceptions concerning this parallel approach. First, it is recommended to start your inductive unknown-unknown experiments [= Red Team/Cell], only after you have carried out your analysis for the other three quadrants. If not, you will infinitely carry out inductive experiments.³⁷ To put it in plain wording: first do your analytic homework thoroughly, before checking if you missed something in the residual category of the unknown-unknown quadrant. Red Team/Red Cell is meant to establish if anything has been overlooked after application of the other techniques. Secondly, if you enter a new research field, e.g. a new mission abroad, it is likely that your databases are not filled yet. The β gap will now be felt mostly. This may hamper the application of some of the techniques within the unknown-known quadrant, especially the ones that work with quantitative [abductive] correlations. Abductive and deductive techniques of the known-unknown quadrant tend to be dominant in that phase.³⁸ As a result, the known-known quadrant then needs extra emphasis to challenge causal connections made over exhaustive data and information.³⁹

The Rumsfeld matrix: illustrations

A research design is influenced by several factors, dealing both with the research question and the organization in which it is carried out. If we look at the methods and techniques used to answer an intelligence research question, it is influenced by three groups of factors. First, the organizational context is of influence – does the process take place within a central or decentralized structure? And is it carried out by analysts that are generalists or specialists? Second, the level of analysis is of influence – is it, for example, a strategic-political or an operational-tactical analysis? Third, the complexity of the issue at hand asks for different degrees of robustness.

In general, intelligence deals with research questions that encompass complex β related issues. You try to select the methods that are as accurate as possible to address the specific question at hand. To do so, the analyst has to ask himself two questions. At first, which methods and techniques are the most optimal for a certain quadrant of the matrix? Second, how do you apply this method or technique in an optimal way?

This last remark seems superfluous, but the application of a method can be seriously affected by bureaucratic and organizational barriers. For example, Red Team/Red Cell – as an unknown-unknown technique – is oriented at the adversary’s modus operandi [AMO] in the first place. By that it is a perpetrator orientated technique. It implies that not an object to be protected should be the starting point of the research. If an intelligence community would put such limitations on Red Team/Red Cell, the experiment will be contrary to the nature of the method. And it will seriously hamper possible findings. If you would limit, for example, your Red Team/Red Cell exercise to one object a time, you will exclude Mumbai style attacks, by Lashkar-e-Taiba in 2008, beforehand. In finding new AMO’s, you could enrich a Mumbai-style Red Team/Red Cell exercise with a new AMO – not used by terrorists yet, but already employed within the military. Such a new AMO could be, for example, terrorists that move around by the earlier mentioned satellite patrol. Such a Mumbai/Satellite Patrol exercise would be hampered by bureaucratic procedures if it orders for attacks-on-one-building-a-time exercises only. But actually you could have been ahead of the terrorists by already being prepared for their next innovation. It is precisely this anticipation that brings the additional value of a Red Team/Red Cell experiment. It is a perpetrator and AMO orientated experiment. Conditions set by an organization that limit the starting points of a technique – in case of Red Team/Red Cell, the AMO – will seriously harm the effectiveness of your research.

Airport security

Let us turn to a practitioner’s situation, and see how the Rumsfeld matrix can actually be filled in. Airport security concerning possible threats by passengers is taken as an example. This can be worked out in different ways. Say, to unveil weak, but actual existing relationships that may pose a threat, the matrix for a certain airport has been composed as follows:

Purely deductive methods – top-down logic – are not applied in Table 4, and may lead to a gap. This gap can be filled by adding deductive methods as a Fault Tree Analysis [for example, concerning possibilities of smuggling explosives through a detector] or a Quantitative Intrusion Path Analysis [for example, to assess to what extent vital centres of the airport are protected against attacks].

Table 4. The Rumsfeld matrix: airport security & passengers

Data To retrieve	Known	Unknown
Known	Abductive Reasoning Belief Revision2 (Key Assumption Check)	mainly Abductive Reasoning Predictive Profiling Security Questioning
Unknown	Abductive reasoning Data mining: e.g. CAPPSII	Inductive Reasoning Red Team/Red Cell

²Belief Revision can, for example, be used to revise someone’s position as suspect on CAPPSII (Computer Assisted Passenger Pre-screening System II)

⁴⁰

Nuclear Security Summit [NSS]

Now, three matrices are presented in which deductive techniques are included also. As these techniques have a preference for the known-unknown quadrant, the next illustrations are worked out in detail for this quadrant only. The organisation of a large international summit is taken as a starting point, such as, for example, a Nuclear Security Summit [NSS] (See Table 5).

Table 5. Summit: perpetrator analysis

Data To retrieve	Known	Unknown
Known	Abductive: Belief Revision, Key Assumption Check	Deductive/Abductive: SLEIPNIR, SWOT- analysis, CARVER & Shock
Unknown	Abductive: Data mining (SocMInt/Sigint)	Inductive: Red Team/Red Cell

⁴¹

Such a summit will attract a range of different groups from different countries that may influence the security or the mobility of the participants of the summit. This can range from legal law-abiding demonstrations, to black-block rioters, or even terrorist attacks – because of the presence of a large number of leaders. For such a summit, different sub-questions need to be covered, like the main convention centre where opponents need to be kept outside, and its [external] communication that must always be kept intact in order to ask for [external] assistance. If you would compose for these different sub-issues Rumsfeld matrices, you will end up with different research designs.

You would probably start with an analysis of the possible perpetrators. Of the groups you think may try at disturbing the summit, you can make a perpetrator analysis. They may be so numerous, that you want to rank them for their possible impact, in order to select the most dangerous ones from the different categories mentioned. To rank the groups in terms of their relative capabilities, limitations and vulnerabilities, SLEIPNIR⁴² can be used. You can assess for each group their weak and strong points, and the opportunities and weaknesses they will encounter, with a SWOT-inventory. And as the perpetrators can choose between multiple targets,⁴³ a CARVER + Shock analysis⁴⁴ can be carried out.

As a result of this perpetrator analysis, you will very likely be pointed to the need of additional measures such as infosec and opsec.⁴⁵

In order to prevent rioters and terrorists to enter the convention centre, you will lay rings of barriers – physically and through security guards/police officers – around and in the complex. Rioters and terrorists will use different AMO’s to enter the centre. So, a technique must be chosen to calculate the time needed to take the barriers for the different AMO’s. Different techniques can be used. You may opt for a Fault Tree Analysis⁴⁶ to evaluate the AMO’s. Yet, Quantitative Intrusion Path Analysis⁴⁷ seems to be more suitable to calculate the delay at the security rings opponents have to pass.⁴⁸

To keep the external communication intact – vital in cases of an emergency – a Fault Tree Analysis seems to be one of the more likely methods. It means that for the second question – to protect the convention centre – a different deductive technique is used than for the third question – to keep its external communication intact. It is illustrated in the next two Tables 6 and 7.

Table 6. Summit & Convention Centre: to keep opponents out of the complex

Data To retrieve	Known	Unknown
Known	Abductive: Key Assumption Check, Peer Review Techniques	mainly Deductive (main frame): Quantitative Intrusion Path Analysis (quantitative)
Unknown	Abductive: Data mining (AMO’s)^3	Inductive: Red Team/Red Cell

³In the Netherlands, the company Sosecure initiated a data base for AMO’s. A finding concerned the theft of copper. First, large numbers of copper cables were stolen from the railway company. After additional security measures, the gangs moved to theft of copper in the public space [art]. Then, additional measures were taken there. A waterbed mechanism developed that was discovered by the data base of Sosecure (Leen Van Der Plas, interview by Giliam de Valk, April 10, 2018.)

Table 7. Summit & Convention Centre: to keep the communication intact

Data To retrieve	Known	Unknown
Known	Abductive: Key Assumption Check, Peer Review Techniques	Deductive: Fault Tree Analysis (qualitative and quantitative)
Unknown	Abductive: Data mining (communications breakdown)	Inductive: Red Team/Red Cell

Limitations of the application of the Rumsfeld matrix

The composition of the eventual research design depends very much on your exact question. And each question will dictate the methods and techniques needed to answer it. As a result, special attention is needed to be paid to the formulation of the research question.⁴⁹

As the research question will influence the composition of the matrix, the matrix will – in turn – influence the way an analysis is carried out. This also applies to the way, for example, a Red Team/Red Cell experiment is executed. Red Team/Red Cell in the context of Tables 6 and 7 will very much be orientated at only one complex – the Convention Centre. But, as stated before in this section, if Red Team/Red Cell is limited to one object, it will not encompass, for example, a Mumbai/Satellite Patrol Red Cell experiment. Only in this first table – the perpetrator perspective – this will be an option. If this first sub-question of Table 5 would be excluded from the Red Cell group, it would harm the scope of such an experiment – although the Red Cell is correctly listed in each of the other two matrices. Therefore, Red Team/Red Cell should not be limited by sub-questions, because otherwise they will certainly overlook unknown-unknowns. So, the Rumsfeld matrix is meant as a help for a research design for a specific [sub-]question of an issue. It is definitely not meant for to work the other way around – limiting the scope and application of techniques – and therefore a wrong application of the methodology. Especially for groups that execute a Red Team/Red Cell experiment, it must be guaranteed that they can set their own agenda and formulate their own research questions.⁵⁰

Conclusion

At the NISA conference, the central focus was on the analyst and the analysis. As explained, intelligence analysis differs from science. The differences encompasses how it is dealt with theory, methodology, and logic. It calls for new and additional tooling.

The Rumsfeld matrix seems to be promising to cope with some issues of the β gap. The Rumsfeld matrix can be used for complex questions that will then be split up in sub-questions. For each of those sub-questions it is established which methods and techniques are the best to answer that sub-question. By applying the matrix on complex problems, it may reduce – at least at a methodological level – the margins that you do not discover a weak, but actual existing, relationship [leading to a threat] between phenomena [= β]. Issues may lose a part of the ‘wickedness’ of their problems as a result of the application of the Rumsfeld matrix.

However, the Rumsfeld matrix should not be used too rigidly. Especially concerning possible gaps that start from the unknown [Unknown-Unknown and Unknown-Known] it is particularly recommended that [sub-]research questions should not limit the application of inductive techniques. Otherwise, unknowns can easily be overlooked. So, the Rumsfeld matrix is an instrument to select the best methods and techniques for a certain [sub-]question, but it is not meant to limit the execution of those [inductive] techniques. A precondition is, therefore, a thorough training of intelligence practitioners in methodology, methods and techniques.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Giliam de Valk

In 2005, Giliam de Valk published his PhD on the quality intelligence analyses have to meet. He is specialized in the methodology of security and intelligence analysis. He has worked at the University of Amsterdam, the University of Utrecht, and the Netherlands Defense Academy where he coordinated and lectured a minor on intelligence studies. At the moment he is an assistant professor at the Institute for Security and Global Affairs, Leiden, Leiden University

Onno Goldbach

Onno Goldbach After finishing his Master’s degree in Physical Geography, Onno Goldbach joined the Royal Netherlands army as a geospatial analyst. During his service at the Dutch Defense Intelligence & Security Institute, Harde, he met academic counterparts. One of them was Giliam de Valk, with whom he surveyed and worked out innovative ideas. This article is one of these ideas.

Notes

¹ A. D De Groot, Methodologie: Grondslagen Van Onderzoek En Denken in De Gedragswetenschappen (Assen: Van Gorcum, 1994), 42, 99.

² P. J. Van Strien, Praktijk Als Wetenschap: Methodologie Van Het Sociaal-wetenschappelijk Handelen (Assen: Van Gorcum, 1986), 56–58.

³ Van Strien, Praktijk Als Wetenschap, 18–19.

⁴ Yin, Case Study Research: Design and Methods (Los Angeles: Sage, 2014).

⁵ Guillaume Gustav de Valk, “Dutch Intelligence – Towards a Qualitative Framework for Analysis” (PhD diss., University of Groningen, 2005) (Den Haag: Boom Juridisch), 66–67.

⁶ The need to arrange methods and techniques for their β capabilities was first referred to by Giliam de Valk, “Effectiviteit vanuit methodologisch perspectief: welke gevolgen heeft de introductie van nieuwe methoden en technieken?” in Contraterrorisme en ethiek, eds. Michael Kowalski and Martijn Meeder (Amsterdam: Boom, 2011), 69–82.

⁷ Giliam de Valk, “All-source intelligence,” in Inlichtingen- en veiligheidsdiensten, eds. Beatrice de Graaf, Erwin Muller, and Joop van Reijn (Alphen aan den Rijn: Kluwer, 2010), 530–1.

⁸ There are handbooks on, for example, Red Teaming. In some handbooks, specialized techniques are also dealt with, for example: Richards J. Heuer and Randolph H. Pherson, Structured Analytic Techniques for Intelligence Analysis (Washington, DC: CQ Press, 2011), §5.7 & §9.6, 122–9, 263–4.

⁹ Onno Goldbach of the Ministry of Defense and Giliam de Valk at the then Ad de Jonge Centrum, Institute for Interdisciplinary Studies, University of Amsterdam.

¹⁰ ”DoD News Briefing – Secretary Rumsfeld and Gen. Myers,” United States Department of Defense, February 12, 2002, https://archive.defense.gov/Transcripts/Transcript.aspx?TranscriptID=2636.

¹¹ Geoffrey K. Pullum, “Language Log,” Language Log: No Foot in Mouth, December 2, 2003, http://itre.cis.upenn.edu/~myl/languagelog/archives/000182.html (accessed January 30, 2012).

¹² John Kemp, “Reuters Market Analyst – For Commodities Now,” Commodities Now, March 2011, http://www.commodities-now.com/reports/general/5522-risk-uncertainty-and-black-swans.html (accessed 2012). Kemp refers to Frank Knight – Risk, Uncertainty and Profit, 1921 – and Knightian Uncertainty to explain that economists and insurers have long distinguished between risk and uncertainty. For a matrix representation, see: http://www.ecodigerati.com/content/articles/?page_id=42 (accessed 2012).

¹³ Robert Ahdieh, “Unknowns Unknowns, Uncertainty, Contracts and Crisis,” in Review of On Uncertainty, Ambiguity, and Contractual Conditions, eds. Eric L. Talley, Jotwell, January 24, 2010, https://corp.jotwell.com/unknown-unknowns-uncertainty-contracts/.

¹⁴ Richard C. Walton, “Rumsfield Matrix (Part 1),” http://pdmicrex.blogspot.nl/2010/08/rumsfeld-matrix-part-1.html.

¹⁵ An abstract of the explanation of these axes has appeared in: Giliam de Valk, “Case Studies into the Unknown – Logic & Tooling,” Romanian Intelligence Studies Review Vol. . 21 (2019): 243–68. It was presented at an IAFIE conference in 2019 – after the 2016 NISA conference for which this article was presented, and in which the full and elaborate methodology is explained.

¹⁶ The composition of a β research design with the Rumsfeld matrix is since 2013 part of the Minor Intelligence Studies, first at the University of Amsterdam [Ad de Jonge Centre] and, since 2017, at the University of Leiden [ISGA].

¹⁷ Data: general data, including non-issue specific. Information: issue-specific data. Knowledge: processed and tested information.

¹⁸ Generic Early Warning Handbook, Report, EAPC/Council Operations and Exercise Committee (NATO, 2001), 1–97.

¹⁹ See, for example, A Military Guide to Terrorism in the Twenty-First Century, Handbook, US Army Training and Doctrine Command, August 15, 2007, https://fas.org/irp/threat/terrorism/guide.pdf.

²⁰ For an explanation, see heading ‘Different classes of reasoning.’

²¹ Jürgen Simon and Jürgen Taeger, Rasterfahndung Entwicklung, Inhalt Und Grenzen Einer Kriminalpolizeilichen Fahndungsmethode (Baden-Baden: Nomos-Verl.-Ges., 1981). To cope with terrorist groups as the Rote Armee Fraktion, the BKA developed typical unknown-known techniques as Rasterfahndung and Schlebnetzfahndung. It would be interesting to assess if a certain education, company culture, or character structure will lead to a preference to use only certain quadrants of the matrix.

²² Osama Bin Laden’s special operations man, Ali Mohammed, for example, obtained information on unconventional warfare, counterinsurgency operations, and how to command elite soldiers on difficult missions. He was an assigned sergeant with the U.S. Army Special Operations – and unofficially an assistant instructor at the JFK Special Operations Warfare School – at Fort Bragg, North Carolina. He marked some documents as ‘Top Secret for Training otherwise unclassified’ (Steven Emerson, “Osama Bin Laden’s Special Operations Man,” Journal of Counterterrorism & Security International (Fall 1998)).

²³ The Red Team Handbook, Report, University of Foreign Military and Cultural Studies, April 2012, https://usacac.army.mil/sites/default/files/documents/ufmcs/The_Red_Team_Handbook.pdf.

²⁴ The British Military devised this technique. In standard patrol flankers, units inspect possible ambush points or dead space areas. For satellite patrol, this is further developed, and intentionally separates itself visually and physically from the base unit of the patrol, outside the visual contact. It demands a better communication and is more difficult to command and control (Urban Operations III: Patrolling. Student Handout. Marine Corps Training Command, https://www.trngcmd.marines.mil/Portals/207/Docs/TBS/B4R5579XQ-DMUrban Operations III – Patrolling.pdf?ver=2016-02-10-114414-840). If terrorists would adapt satellite patrol, they probably would be able to neutralize a series of routine police road blocks.

²⁵ In research, robustness refers to applying several methods and techniques in an analysis. The more such independent tests are performed with a positive outcome, the more plausible the conclusion will be. Consequently, the finding does not depend on the analytical method used. To apply many methods to the same set of data, the margin of error is reduced (Guillaume Gustav de Valk, “Dutch Intelligence – Towards a Qualitative Framework for Analysis: With Case Studies on the Shipping Research Bureau and the National Security Service (BVD),” (PhD diss., University of Groningen, 2005) (Den Haag: Boom Juridisch), 67–68. https://www.rug.nl/research/portal/files/33123437/c9.pdf). It is assumed here that the error margin is not only reduced by applying more methods, but also by applying different classes of methods. There is another reason that calls for robustness of the research design as well. Intelligence deals with events that are sometimes very hard, or even impossible, to predict (‘black swans’). This points to the limitations of knowledge and assessing the future based on the past. It should cause every analyst to question assumptions and test the robustness of its research design. And you should not simply argue ‘after the event that some things were simply unforecastable’ (Kemp, “Reuters Market Analyst”).

²⁶ Two additional remarks need to be made here. First, a well-known danger of inductive logic is illustrated by the observation of the first 100 swans: just because they happen to be white does not mean that all swans are white. Second, mathematical induction, used for mathematical proof, is here arranged as a deductive form of reasoning: in inductive reasoning, the conclusion can be false.

²⁷ Rosa Voulon, Handboek Analyse. Theorievorming En Methodologie in Inlichtingenanalyse (‘t Harde: Defensie Inlichtingen En Veiligheids Instituut, 2009), 24–27. Grabo mentions these three approaches – deduction, induction, and abduction – in relation to the analysis of indications. She puts that it almost always will be a process of abduction (= inference) (Cynthia M. Grabo, Anticipating Surprise: Analysis for Strategic Warning (Lanham: University Press of America, 2004), 42–43).

²⁸ De Groot, Methodologie, 38, 76–82, 38; Grabo, Anticipating Surprise, 42–44; and Voulon, Handboek Analyse, 24–27.

²⁹ In such experiments, the unknown-unknown–is addressed, by speaking of a residual threat. Practitioners use so-called Red Team and Red Cell experiments to reduce the residual threats. It deviates from scientific experiments in which a hypothesis is tested – and, by that, is related to the α.

³⁰ Giliam de Valk and Willemijn Aerdts, “Inlichtingenwerk Vanuit Een Methodologisch Perspectief,” Justitiële Verkenningen 44, no. 1 (2018): 120–122.

³¹ Giliam de Valk, “Red Team and Science” (Presentation for De Nederlandsche Bank (DNB), Den Haag, June 8, 2012).

³² Richard Heuer, “Biases in Evaluation of Evidence,” Studies in Intelligence (Winter 1981) Box 8, 92-3: 31–35.

³³ ‘This process is as follows: if you receive evidence, you postulate a set of causal connections that explains this evidence. The stronger you perceive the relation between facts leading to that causal connection; the stronger you perceive that causal connection. This attribution tends to persist even after the evidence that created those connections has been fully discredited. Even if you learn that the information – on which you developed your causal connection – comes from an uncontrolled source who may be trying to manipulate you, this does not necessarily reduce the impact of this causal connection. In general, the ‘early but incorrect impression tends to persist because the amount of information necessary to invalidate a hypothesis is considerably greater than the amount of information required to make an initial interpretation […] People form impressions on the basis of very little information, but once formed, they do not reject or change them unless they obtain rather solid evidence’ From: de Valk, Dutch Intelligence, 82. De Valk is referring here to Heuer, ‘Biases in Evaluation of Evidence.”

³⁴ To derive conclusions from possible hypotheses, and to invert conditionals to apply probabilistic abduction.

³⁵ For the mentioned techniques, see: Heuer and Pherson, Structured Analytic Techniques, 140–3, 163–6.

³⁶ Arthur Hulnick, “The Intelligence-Producer-Policy Consumer Linkage,” Studies in Intelligence (Winter 1985) Box 9, 108-7: 76–9.

³⁷ See, the Table 2.

³⁸ You will try to get your data bases filled in an optimal way by starting to monitor those aspects or places that will yield the most relevant information. For this, GEOINT or a theoretical steered approach such as the choke point theory [this is a so-called B-theory] can be made use of (Giliam de Valk, “All-source Intelligence,” 507–33, 527–8).

³⁹ As put, this is needed, among others, for the persistence of impressions based on discredited evidence in the causal connection (Heuer, “Biases in Evaluation of Evidence,” 44–46).

⁴⁰ For an explanation, see further down this section.

⁴¹ Nuclear Security Summits (NSS) are originally an initiative of President Obama. In 2010 he invited state and government leaders of a large number of countries in order to cope the threat of nuclear terrorism, and to fight illegal trade in nuclear material “Uw Verzoek Inzake Top Nucleaire Veiligheid 2014 (NSS 2014/2019),” Frans Timmermans to Tweede Kamer, April 26, 2013.

⁴² SLEIPNIR is ‘an analytical technique developed to rank order organized groups of criminals in terms of their relative capabilities, limitations and vulnerabilities. The rank ordered lists of groups are components of strategic intelligence assessments used to recommend intelligence and enforcement priorities’. To cope with organized crime, for example, attributes are selected as (in rank): 1 corruption; 2 violence; 3 infiltration; 4 expertise; 5 sophistication; 6 subversion; 7 strategy; 8 discipline; 9 insulation; 10 intelligence use; 11 multiple enterprises; 12 mobility; 13 stability; 14 scope; 15 monopoly; 16 group cohesiveness; 17 continuity; 18 links to other organized crime groups; 19 links to criminal extremist groups. For terrorism, a different attribute set is made (Steven J. Strang, Project SLEIPNIR: An Analytical Technique for Operational Priority Setting (Ottawa: RCMP, 2019), 1–5).

⁴³ Because of the size of such a summit, different locations to meet will be used, as will participants dine and sleep at different locations. Not only these locations can be targeted, but also the supply and transportation lines of a summit. As opponents can choose between multiple targets, an analysis is needed of the target selection from the perpetrator’s perspective.

⁴⁴ CARVER + Shock is an acronym for criticality (measure of impact of an attack), accessibility (ability to physically access and egress from target), recuperability (ability of system to recover from an attack), vulnerability (ease of accomplishing attack), effect (amount of direct loss from an attack as measured by loss of production) and recognisability (ease of identifying target) + shock (the combined health, economic, and psychological impacts of an attack). It is a prioritization tool, a system of target acquisition, to rank potential targets according to a scale. By identifying and ranking the potential targets, attack resources can be efficiently used. It assesses the vulnerabilities within a system, industry, or infrastructure. Originally developed for US special forces, it is now also applied by, among others, the US Food and Drug Administration to enhance ‘food defence’ Consumer Updates, accessed 2012, http://www.fda.gov/ForConsumers/ConsumerUpdates/ucm094560.htm; and accessed 2012, http://www.fda.gov/Food/NewsEvents/ConstituentUpdates/ucm180608.htm.

⁴⁵ In Red Team/Red Cell exercises the University of Amsterdam carried out, infosec and opsec was a returning point of attention. Sometimes, technical personnel – crucial for the maintenance of the infrastructure – complained on social media about their managers. In other cases, the target organisation had opsec and infosec intact, but it others, like local municipalities and provinces, published sensitive material on their sites that could be used to plan a terrorist attack.

⁴⁶ In 1962, H. A. Watson of Bell Telephone Laboratories developed the Fault Tree Analysis – also referred at as Event Tree Analysis – for the US Air Force (Minutemen). It is a logic diagram to relate conditions that precede faults and undesired events. At the top of the schedule, the undesired event – end state – is placed. It can be applied in both a qualitative and quantitative way: Anna L. Martensen and Ricky W. Butler, The Fault-Tree Compiler (Hampton, VA: National Aeronautics and Space Administration, Langley Research Center, 1987), 1–3, 6–9.

⁴⁷ The Quantitative Intrusion Path Analysis is a method that is known under different synonyms – with many different variants – often referring to the name of the specific software that is used to carry it out. It is designed not only to weigh physical security measures, but also the human factor. Thus, it could be measured if an opponent could enter – and at what speed, by what AMO – secured critical infrastructure. It measures the delay by physical barriers, and calculates issues as recognition, warning and reaction time (for a full system case study, see, for example: PR&PP Evaluation: ESFR Full System Case Study Final Report, Report, Proliferation Resistance and Physical Protection Evaluation Methodology Working Group, October 2009, https://www.gen-4.org/gif/upload/docs/application/pdf/2013-09/prpp_csreport_and_appendices_2009_10-29.pdf). It is also worked out in several variations for cyber.

⁴⁸ A Quantitative Intrusion Path Analysis (QIPA) would also have been helpful in case of, for example, the art robbery at the Rotterdam Kunsthal. In October 2012, a gang robbed paintings – and later partly burned these – by Claude Monet, Pablo Picasso, Henri Matisse, Paul Gauguin, Meyer de Haan and Lucian Freud, worth many millions of euros. Although there was, rightly so, a lot of criticism on the poor state of the locks on the doors, the robbery was an analytic failure in the first place. There was an alarm, but you should have calculated the delay that each security ring would have confronted the robbers with. Then you combine it with the time the reinforcement (police/security) needs to be in place. Only by such a combined analysis, a sound security plan can be designed. And QIPA is per excellence a technique to calculate that.

⁴⁹ In the intelligence literature, some publications deal in-depth with the issue of the formulation of the research question, for example: Heuer and Pherson, Structured Analytic Techniques, § 4.3; and Brian Manning and Kristan Wheaton, “Making “Easy Questions” Easy: The Difficulty of Intelligence Requirements,” International Journal of Intelligence and Counterintelligence 26, no. 3 (September 2013): 597–611.

⁵⁰ The logic of this freedom of formulating the research question lies in the type of quadrant – the unknown-unknown. In practice, at the Ad de Jonge Centre such limitations were felt as a result of the scope set by the authorities.