The efficacy of ACH in mitigating serial position effects and confirmation bias in an intelligence analysis scenario

ABSTRACT

An experimental study was conducted to test whether the version of the structured analytical method Analysis of Competing Hypotheses (ACH) taught by the Cabinet Office to the United Kingdom’s intelligence community provides effective mitigation of the cognitive biases of serial position effects and confirmation bias in an intelligence analysis. ACH had no statistically significant mitigative impact on the proportion of participants that exhibited serial position effects or confirmation bias, or the impact of confirmation bias on the analytical process. The most significant factor that influenced participants’ judgements of the credibility of information was the possibility of deception or dishonesty.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. Tversky and Kahneman, ‘Judgement Under Uncertainty’.

2. Hendrick and Costantini, Number Averaging Behaviour.

3. Hendrick et al., Attention Decrement.

4. Chapman and Chapman, ‘Genesis of Popular Observations’.

5. Johnson, National Security Intelligence.

6. This has colloquially been framed as the ‘art versus science’ debate.

7. Richards, Art and Science of Intelligence Analysis, and; Treverton and Fishbein, Making Sense of Transnational Threats. Other academics have argued that biases can be overcome by intensifying the level of knowledge you have of your adversary (Handel 1984).

8. Heuer and Pherson, Structured Analytical Methods.

9. Ibid.

10. Those that do tend to focus on applying systems analysis programmes to threat prediction.

11. This can be seen from the analysis of several internal training manuals from the UK and US intelligence and defence communities that are now available in open source: Cabinet Office 2015; United Nations Office on Drugs and Crime 2011; Grabo 2010; United States Government 2009; Criminal Intelligence Service Canada 2007; Krizan 1999; Taylor 1990, and; Kent 1949.

12. Jones, ‘Critical Epistemology’.

13. Ibid.

14. Cheikes et al. Confirmation Bias in Complex Analyses.

15. Observed effect size (Cohen’s d) = 0.8, probability level 0.05, observed power: 0.78034989. This means that the chance of type 1 error is 5% and the chance of type 2 error is 32%. Observed effect size (Cohen’s d) = 0.7, probability level 0.05, observed power: 0.68121420. Observed effect size (Cohen’s d) = 0.5, probability level 0.05, observed power 0.44691462. Observed effect size (Cohen’s d) = 0.3, probability level 0.05, observed power: 0.23025184.

16. The Bayesian prior for each participant were set before any information was processed as uninformative priors. A Bayesian prior expresses the degree of belief of an individual between different outcomes before any information has been processed. Uninformative priors reflect a balance among outcomes before information has been taken to account. For the study, the uninformative priors were set at 50/50 division of degree of belief scores allocated between Hypothesis 1 and Hypothesis 2.

17. Details from the intelligence case available to the United States (US) intelligence community were selected to develop an intelligence analysis scenario for the study that has ecological validity. The adaption of this intelligence case drew principally upon information provided in the 2005 US government official review into weapons of mass destruction (WMD) in Iraq by the Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction. Part of the report’s remit included an investigation into the accuracy of intelligence on Iraqi WMD up to March 2003. The report provided a sufficient level of detail to allow a suitably adapted version of the intelligence case to be provided. Some additional details were also obtained from Drogin (2008). The 2004 House of Commons official review of the intelligence on weapons of mass destruction was also reviewed. However, it supplied no additional information on the intelligence case not already provided by the US official review and Drogin (2008).

18. The ACH version delivered by the PHIA to the ACH group in this study solely uses diagnostic value as the scoring system. Participants were also asked to provide assessments of credibility. However, the credibility scores were not used to calculate the overall ACH scores, and no guidance was provided on how credibility of information should be determined.

19. The test had to be conducted with a sample that contained outliers. The outliers were caused by the ordinal nature of the variables and the relatively small sample sizes of the research groups. This may have impacted the validity of the two-way ANOVA but retained the integrity of the sample.

20. One participant in the sample provided split category answers. As multinomial logistic regression test assumes that all categories in the nominal and ordinal variables are mutually exclusive, this participant’s results were not included. Due to the ordinal nature of variables involved, and the relatively small sample sizes of the research groups, a comparatively large number of outliers were detected in isolated cases.

21. As the serial orders were consistent at both points A and B in information processing, where an effect consistent with primacy was observed, statistical analysis for main and interaction effects cannot be used to determine whether serial order was the cause.

22. This study has also been republished under the name of Lehner et al. (2008).

23. The study used the same ACH procedure as used by Folker (2000).

24. Schwoebel, Explosion Aboard the Iowa, and; Thompson, A Glimpse of Hell.

25. Cheikes et al. (2004) described the primacy effect as a form of confirmation bias, manifesting in a weighting effect. However, the effect is more consistent with primacy. Confirmation bias would manifest separately from primacy, although confirmation bias could plausibly be the result of a primacy effect.

26. The version of ACH taught by the PHIA to the UK’s Intelligence Community uses a semantic scoring scale for diagnostic value, whereas the version used by Cheikes et al (2004) used an integer scale. However, the two versions are compatible and, therefore, the results of the two studies are comparable.

27. Multiple published studies into serial position effects and confirmation bias have included sample sizes of below 50 participants. Key examples include: Cheikes et al. (2004); Koslowski and Maqueda (1993); Mynatt, Doherty and Tweney (1997); Wason (1960); Tolcott et al. (1989b); Perrin et al. (2001); Anderson (1964); Anderson (1965b); Anderson (1967); Anderson (1973a); Anderson and Barrios (1961), and; Asch (1946). However, it does not follow that these studies lacked sufficient statistical power.

Additional information

Notes on contributors

Martha Whitesmith

Martha Whitesmith completed a PhD in Intelligence and International Security at King’s College London in July 2018. She holds an MA in Philosophy from the University of London.