ODNI as an analytic ombudsman: is Intelligence Community Directive 203 up to the task?

Bibliography

• Artner, S., R. Girven, and J. Bruce. Assessing the Value of Structured Analytical Techniques in the U.S. Intelligence Community. Santa Monica, CA: RAND Corporation, 2016. https://www.rand.org/pubs/research_reports/RR1408.html.
   Google Scholar
• Betts, R. “Analysis, War, and Decision: Why Intelligence Failures are Inevitable.” World Politics 31, no. 1 (1978): 61–89. doi:10.2307/2009967.
   Web of Science ®Google Scholar
• Cardillo, R. “A Cultural Evolution.” Studies in Intelligence 54, no. 3 (2010): 43–49.
   Google Scholar
• Chang, W., E. Berdini, D. R. Mandel, and P. E. Tetlock. “Restructuring Structured Analytic Techniques in Intelligence.” Intelligence and National Security 33, no. 3 (2018): 337–356. doi:10.1080/02684527.2017.1400230.
   Web of Science ®Google Scholar
• Cicchetti, D. “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instrument in Psychology.” Psychological Assessment 6 (1994): 284–290. doi:10.1037/1040-3590.6.4.284.
   Google Scholar
• Coulthart, S. “An Evidence-Based Evaluation of 12 Core Structured Analytic Techniques.” International Journal of Intelligence and CounterIntelligence 30, no. 2 (2017): 368–391. doi:10.1080/08850607.2016.1230706.
   Web of Science ®Google Scholar
• Fingar, T. Analytic Transformation: Unleashing the Potential of a Community of Analysts. Washington, DC: Office of the Director of National Intelligence, 2008. Accessed May 10, 2018. https://www.hsdl.org/?abstract&did=29867.
   Google Scholar
• Fingar, T. Reducing Uncertainty: Intelligence Analysis and National Security. Stanford: Stanford Security Studies, 2011.
   Google Scholar
• Fingar, T. Keynote Address to the 2008 INSA Analytic Transformation Conference Orlando, Florida. Accessed September 4, 2008. https://www.dni.gov/files/documents/Newsroom/Speeches%20and%20Interviews/20080904_speech.pdf.
   Google Scholar
• Friedman, J. A., and R. Zeckhauser. “Assessing Uncertainty in Intelligence.” Intelligence and National Security 27, no. 6 (2012): 824–847. doi:10.1080/02684527.2012.708275.
   Google Scholar
• Friedman, J. A., and R. Zeckhauser. “Why Assessing Estimative Accuracy Is Feasible and Desirable.” Intelligence and National Security 31, no. 2 (2016): 178–200. doi:10.1080/02684527.2014.980534.
   Web of Science ®Google Scholar
• Gentry, J. “Intelligence Failure Reframed.” Political Science Quarterly 123, no. 2 (2008): 247–270. doi:10.1002/j.1538-165X.2008.tb00624.x.
   Web of Science ®Google Scholar
• Gentry, J. “Has the ODNI Improved U.S. Intelligence Analysis?” International Journal of Intelligence and CounterIntelligence 28, no. 4 (2015): 637–661. doi:10.1080/08850607.2015.1050937.
   Web of Science ®Google Scholar
• Heuer, R. J., and R. H. Pherson. Structured Analytic Techniques for Intelligence Analysis. Washington, DC: CQ Press, 2010.
   Google Scholar
• IARPA. Broad Agency Announcement: Crowdsourcing Evidence, Argumentation, Thinking and Evaluation. Washington, DC: Office of Anticipating Surprise, 2016. Accessed March 3, 2018. https://www.fbo.gov/index?tab=documents&tabmode=form&subtab=core&tabid=b315926fb14cb6d31026476229e920ab.
   Google Scholar
• Immerman, R. H. “Transforming Analysis: The Intelligence Community’s Best Kept Secret.” Intelligence and National Security 26, no. 2–3 (2011): 159–181. doi:10.1080/02684527.2011.559138.
   Google Scholar
• Immerman, R. H. “Intelligence and the Iraq and Afghanistan Wars.” Political Science Quarterly 131 (2016): 477–501. doi:10.1002/polq.v131.3.
   Web of Science ®Google Scholar
• Intelligence Community Directive (ICD) 203. Analytic Standards. Washington, DC: Office of the Director of National Intelligence, 2015.
   Google Scholar
• Jonsson, A., and G. Svingby. “The Use of Scoring Rubrics: Reliability, Validity and Educational Consequences.” Educational Research Review 2, no. 2 (2007): 130–144. doi:10.1016/j.edurev.2007.05.002.
   Google Scholar
• Judd, T. P., C. Secolsky, and C. Allen. “Being Confident about Results from Rubrics.” National Institute for Learning Outcomes Assessment Viewpoint Blog. 2012. http://illinois.edu/blog/view/915/68373?displayOrderdesc&displayTypenone&displayColumncreated&displayCount1.
   Google Scholar
• Kent, S. “Words of Estimative Probability.” Studies in Intelligence 8, no. 4 (1964): 49–65.
   Google Scholar
• Lowenthal, M. “Towards a Reasonable Standard for Analysis: How Right, How Often on Which Issues?” Intelligence and National Security 23, no. 3 (2008): 303–315. doi:10.1080/02684520802121190.
   Google Scholar
• Lowenthal, M. “A Disputation on Intelligence Reform and Analysis: My 18 Theses.” International Journal of Intelligence and CounterIntelligence 261 (2012): 31–37.
   Google Scholar
• Marchio, J. “Analytic Tradecraft and the Intelligence Community: Enduring Value, Intermittent Emphasis.” Intelligence and National Security 29, no. 2 (2014): 159–183. doi:10.1080/02684527.2012.746415.
   Google Scholar
• Marrin, S. “Training and Educating US Intelligence Analysis.” International Journal of Intelligence and CounterIntelligence 22, no. 1 (2009): 131–146. doi:10.1080/08850600802486986.
   Google Scholar
• Marrin, S. “Evaluating the Quality of Intelligence Analysis: By What (Mis) Measure?” Intelligence and National Security 27, no. 6 (2012): 896–912. doi:10.1080/02684527.2012.699290.
   Google Scholar
• Marrin, S., and J. Clemente. “Improving Intelligence Analysis by Looking at the Medical Profession.” International Journal of Intelligence and CounterIntelligence 18, no. 4 (2005): 707–729. doi:10.1080/08850600590945434.
   Google Scholar
• Marrin, S., and J. Clemente. “Modelling and Intelligence Analysis Profession on Intelligence.” International Journal of Intelligence and CounterIntelligence 19, no. 4 (2006/2007): 642–645. doi:10.1080/08850600600829882.
   Google Scholar
• Patterson, E., S. McNee, D. Zelik, and D. Woods. “Insights from Applying Rigor Metric to Healthcare Incident Investigations.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 52, no. 21 (2008): 1766–1770. doi:10.1177/154193120805202116.
   Google Scholar
• Phythian, M. “The Perfect Intelligence Failure? U.S. Pre-War Intelligence on Iraqi Weapons of Mass Destruction.” Politics & Policy 34 (2006): 400–424. doi:10.1111/j.1747-1346.2006.00019.x.
   Google Scholar
• Reddy, Y. M., and H. Andrade. “A Review of Rubric Use in Higher Education.” Assessment and Evaluation in Higher Education 35, no. 4 (2010): 435–448. doi:10.1080/02602930902862859.
   Web of Science ®Google Scholar
• RiCharde, R. S. “The Humanities Versus Interrater Reliability.” Assessment Update 20, no. 4 (2008): 10–11.
   Google Scholar
• Silberman, L., and C. Robb. Report to the President of the United States. Washington: USGPO, 2005.
   Google Scholar
• Tetlock, P. E., and B. A. Mellers. “Intelligent Management of Intelligence Agencies: Beyond Accountability Ping-Pong.” American Psychologist 66, no. 6 (2011): 542–554. doi:10.1037/a0024285.
   PubMed Web of Science ®Google Scholar
• Turbow, D. J., and J. Evener. “Norming a VALUE Rubric to Assess Graduate Information Literacy Skills.” Journal of the Medical Library Association : JMLA 104, no. 3 (2016): 209–214. doi:10.3163/1536-5050.104.2.004.
   PubMed Web of Science ®Google Scholar
• U.S. Government. A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis. Washington: Center for the Study of Intelligence Analysis, 2009.
   Google Scholar
• van Gelder, T., and R. de Rozario. “Pursuing Fundamental Advances in Reasoning.” In Artificial General Intelligence, Proceedings of the 10 International Conference, AGI-2017, Lecture Notes in Artificial Intelligence, edited by T. Everitt, A. Potapov, and B. Goertzel, 259–262. Cham: Springer, 2017.
   Google Scholar
• Wohlstetter, R. Pearl Harbor: Warning and Decision. Stanford: Stanford University Press, 1962.
   Google Scholar
• Zegart, A. “September 11 and the Adaptation Failure of U.S. Intelligence Agencies.” International Security 29, no. 4 (2005): 78–111. doi:10.1162/isec.2005.29.4.78.
   Web of Science ®Google Scholar
• Zegart, A. “An Empirical Analysis of Failed Intelligence Reforms before September 11.” Political Science Quarterly 121 (2006): 33–60. doi:10.1002/j.1538-165X.2006.tb00564.x.
   Web of Science ®Google Scholar
• Zelik, D., E. Patterson, and D. Woods. “Understanding Rigor in Information Analysis.” In 8th International Conference on Naturalistic Decision Making. Pacific Grove, 2007.
   Google Scholar
• Zelik, D., E. Patterson, and D. Woods. “Measuring Attributes of Rigor in Information Analysis.” In Macrocognition Metrics and Scenarios: Design and Evaluation of Real-World Teams, edited by E. Patterson and J. Miller, 65–83. Aldershot: Ashgate, 2010.
   Google Scholar
• Zelik, D., E. Patterson, and D. Woods. “Judging Sufficiency: How Professional Intelligence Analysts Assess Analytical Rigor.” In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 51, no. 4, 318–322. Los Angeles: SAGE Publications, 2007.
   Google Scholar