Designing human-AI systems for complex settings: ideas from distributed, joint, and self-organising perspectives of sociotechnical systems and cognitive work analysis

References

• Ashby, W. R. 1956. An Introduction to Cybernetics. New York: John Wiley & Sons.
   Google Scholar
• Ashoori, M., and C. Burns. 2010. “Reinventing the Wheel: Control Task Analysis for Collaboration.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 54 (4): 274–278. doi:10.1177/154193121005400402.
   Google Scholar
• Asmayawati, S., and J. Nixon. 2020. “Modelling and Supporting Flight Crew Decision-Making during Aircraft Engine Malfunctions: Developing Design Recommendations from Cognitive Work Analysis.” Applied Ergonomics 82: 102953. doi:10.1016/j.apergo.2019.102953.
   PubMed Web of Science ®Google Scholar
• Bainbridge, L. 1983. “Ironies of Automation.” Automatica 19 (6): 775–779. doi:10.1016/0005-1098(83)90046-8.
   Web of Science ®Google Scholar
• Bennett, K. B., and J. M. Flach. 2011. Display and Interface Design: Subtle Science, Exact Art. Boca Raton, FL: CRC Press.
   Google Scholar
• Bennett, K. B., S. M. Posey, and L. G. Shattuck. 2008. “Ecological Interface Design for Military Command and Control.” Journal of Cognitive Engineering and Decision Making 2 (4): 349–385. doi:10.1518/155534308X377829.
   Google Scholar
• Bigley, G. A., and K. H. Roberts. 2001. “The Incident Command System: High-Reliability Organizing for Complex and Volatile Task Environments.” Academy of Management Journal 44 (6): 1281–1299. doi:10.2307/3069401.
   Web of Science ®Google Scholar
• Bisantz, A. M., E. Roth, B. Brickman, L. L. Gosbee, L. Hettinger, and J. McKinney. 2003. “Integrating Cognitive Analyses in a Large-Scale System Design Process.” International Journal of Human-Computer Studies 58 (2): 177–206. doi:10.1016/S1071-5819(02)00130-1.
   Web of Science ®Google Scholar
• Bisantz, A. M., and K. J. Vicente. 1994. “Making the Abstraction Hierarchy Concrete.” International Journal of Human-Computer Studies 40 (1): 83–117. doi:10.1006/ijhc.1994.1005.
   Web of Science ®Google Scholar
• Bogdanovic, J., J. Perry, M. Guggenheim, and T. Manser. 2015. “Adaptive Coordination in Surgical Teams.” BMC Health Services Research 15 (1): 128. doi:10.1186/s12913-015-0792-5.
   PubMed Web of Science ®Google Scholar
• Brady, A., and N. Naikar. 2022. “Development of Rasmussen’s Risk Management Framework for Analysing Multi-Level Sociotechnical Influences in the Design of Envisioned Work Systems.” Ergonomics 65 (3): 485–518. doi:10.1080/00140139.2021.2005823.
   PubMed Web of Science ®Google Scholar
• Brown, Noam, and Tuomas Sandholm. 2019. “Superhuman AI for Multiplayer Poker.” Science (New York, N.Y.) 365 (6456): 885–890. doi:10.1126/science.aay2400.
   PubMed Web of Science ®Google Scholar
• Brown, T., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. 2020. “Language Models Are Few-Shot Learners.” NeurIPS 33: 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
   Google Scholar
• Burns, C. M. 2018. “Automation and the Human Factors Race to Catch up.” Journal of Cognitive Engineering and Decision Making 12 (1): 83–85. doi:10.1177/1555343417724975.
   Web of Science ®Google Scholar
• Bye, R., and A. Naweed. 2008. “Book Review.” Ergonomics 51 (2): 242–244. doi:10.1080/00140130600971275.
   Web of Science ®Google Scholar
• Calhoun, G. 2022. “Adaptable (Not Adaptive) Automation: Forefront of Human-Automation Teaming.” Human Factors 64 (2): 269–277. doi:10.1177/00187208211037457.
   PubMed Web of Science ®Google Scholar
• Camazine, S., J.-L. Deneubourg, N. R. Franks, J. Sneyd, G. Theraula, and E. Bonabeau. 2001. Self-Organization in Biological Systems. Princeton, NJ: Princeton University Press.
   Google Scholar
• Carroll, J. M., and M. B. Rosson. 1992. “Getting around the Task Artifact Cycle: How to Make Claims and Design by Scenario.” ACM Transactions on Information Systems 10 (2): 181–212. doi:10.1145/146802.146834.
   Web of Science ®Google Scholar
• Christoffersen, K., and D. D. Woods. 2002. “How to Make Automated Systems Team Players.” In Advances in Human Performance and Cognitive Engineering Research: Automation, edited by E. Salas, 1–12. Oxford, UK: Elsevier Science/JAI Press.
   Google Scholar
• Colmer, A. 2021. “Agile Command and Control Insights Paper.” Noetic Group. Defence Science and Technoloy Group Emerging Disruptive Technology Assessment Symposium: https://www.dst.defence.gov.au/sites/default/files/events/documents/Agile%20C2%20Insights%20Paper%20F2.pdf.
   Google Scholar
• Cornelissen, M., P. M. Salmon, D. P. Jenkins, and M. G. Lenne. 2013. “A Structured Approach to the Strategies Analysis Phase of Cognitive Work Analysis.” Theoretical Issues in Ergonomics Science 14 (6): 546–564. doi:10.1080/1463922X.2012.668973.
   Google Scholar
• Dekker, S. W., and D. D. Woods. 2002. “MABA-MABA or Abracadabra? Progress on Human-Automation Coordination.” Cognition, Technology & Work 4 (4): 240–244. doi:10.1007/s101110200022.
   Google Scholar
• Dikmen, M. 2022. “A Cognitive Work Analysis Approach to Explainable Artificial Intelligence in Non-Expert Financial Decision-Making.” Doctoral Thesis, University of Waterloo, University of Waterloo’s Institutional Repository. http://hdl.handle.net/10012/18355.
   Google Scholar
• Elasri, Mohamed, Omar Elharrouss, Somaya Al-Maadeed, and Hamid Tairi. 2022. “Image Generation: A Review.” Neural Processing Letters 54 (5): 4609–4646. doi:10.1007/s11063-022-10777-x.
   Web of Science ®Google Scholar
• Elix, B., and N. Naikar. 2008. “Designing Safe and Effective Future Systems: A New Approach for Modelling Decisions in Future Systems with Cognitive Work Analysis.” 8th International Symposium of the Australian Aviation Psychology Association. VIC, Australia: Australian Aviation Psychology Association.
   Google Scholar
• Elix, B., and N. Naikar. 2021. “Designing for Adaptation in Workers’ Individual Behaviors and Collective Structures with Cognitive Work Analysis: Case Study of the Diagram of Work Organisation Possibilities.” Human Factors 63 (2): 274–295. doi:10.1177/0018720819893510.
   PubMed Web of Science ®Google Scholar
• Endsley, M. R. 2017. “From Here to Autonomy: Lessons Learned from Human-Automation Research.” Human Factors 59 (1): 5–27. doi:10.1177/0018720816681350.
   PubMed Web of Science ®Google Scholar
• Ernst, K., E. Roth, L. Militello, C. Sushereba, J. DiIulio, S. Wonderly, E. Roth, S. Scheff, D. Klein, G. Taylor. 2019. “A Strategy for Determining Optimal Crewing in Future Vertical Lift: Human Automation Function Allocation.” Proceedings of the Vertical Lift Society’s 75th Annual Forum & Technology Display, Vertical Flight Society, May 13–16.
   Google Scholar
• Fitts, P. M., M. S. Viteles, N. L. Barr, D. R. Brimhall, G. Finch, E. Gardner, and S. S. Stevens. 1951. Human Engineering for an Effective Air-Navigation and Traffic-Control System. Columbus, OH: Ohio State University Research Foundation.
   Google Scholar
• Flach, J. M. 2012. “Complexity: Learning to Muddle Through.” Cognition, Technology & Work 14 (3): 187–197. doi:10.1007/s10111-011-0201-8.
   Web of Science ®Google Scholar
• Flach, J. M. 2015. “Situation Awareness: Context Matters. A Commentary on Endsley.” Journal of Cognitive Engineering and Decision Making 9 (1): 59–72. doi:10.1177/1555343414561087.
   Google Scholar
• Gibson, J. J. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin.
   Google Scholar
• Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep Learning (Adaptive Computation and Machine Learning Series). Cambridge: MIT Press Academic.
   Google Scholar
• Hajdukiewicz, J. R., C. M. Burns, K. J. Vicente, and R. G. Eggleston. 1999. “Work Domain Analysis for Intentional Systems.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 43 (3): 333–337. doi:10.1177/154193129904300343.
   Google Scholar
• Haken, H. 1988. Information and Self-Organization: A Macroscopic Approach to Complex Systems. Berlin: Springer.
   Google Scholar
• Hancock, P. A. 2017. “Imposing Limits on Autonomous Systems.” Ergonomics 60 (2): 284–291. doi:10.1080/00140139.2016.1190035.
   PubMed Web of Science ®Google Scholar
• Hassall, M. E., P. M. Sanderson, and I. T. Cameron. 2014. “The Development and Testing of SAfER: A Resilience-Based Human Factors Method.” Journal of Cognitive Engineering and Decision Making 8 (2): 162–186. doi:10.1177/1555343414527287.
   Google Scholar
• He, K., X. Zhang, S. Ren, and J. Sun. 2015. “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.” In 2015 IEEE International Conference on Computer Vision (ICCV), 1026–1034. Washington, DC: IEEE Computer Society. doi:10.1109/ICCV.2015.123.
   Google Scholar
• Heath, C., and P. Luff. 1998. “Convergent Activities: Line Control and Passenger Information on the London Underground.” In Cognition and Communication at Work, edited by Y. Engeström and D. Middleton, 6–129. Cambridge: Cambridge University Press.
   Google Scholar
• Hollnagel, E. 2012. “Coping with Complexity: Past, Present and Future.” Cognition, Technology & Work 14 (3): 199–205. doi:10.1007/s10111-011-0202-7.
   Web of Science ®Google Scholar
• Hollnagel, E. 2022. “Cognitive Systems Engineering: RIP.” Erik Hollnagel. https://erikhollnagel.com/ideas/cognitive-systems-engineering.html
   Google Scholar
• Hollnagel, E., and D. D. Woods. 2005. Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. Boca Raton, FL: CRC Press.
   Google Scholar
• Hutchins, E. 1991. “Organizing Work by Adaptation.” Organization Science 2 (1): 14–39. doi:10.1287/orsc.2.1.14.
   Web of Science ®Google Scholar
• Hutchins, E. 1995. Cognition in the Wild. Cambridge: MIT Press.
   Google Scholar
• Hutchins, E. 1999. “Cognitive Artifacts.” In The MIT Encylopedia of the Cognitive Sciences, edited by R. A. Wilson and F. C. Keil, 126–128. Cambridge: The MIT Press.
   Google Scholar
• Hutchins, E. 2006. “The Distributed Cognition Perspective on Human Interaction.” In Roots of Human Sociality: Culture, Cognition and Interaction, edited by N. J. Enfield and S. C. Levinson, 375–398. London: Routledge.
   Google Scholar
• Hutchins, E. L. 1990. “The Technology of Team Navigation.”In Intellectual Teamwork: Social and Technological Foundations of Cooperative Work, J. Galagher, R. E. Kraut, and C. Egido, 191–221. Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Hutchins, E., and T. Klausen. 1998. “Distributed Cognition in an Airline Cockpit.” In Cognition and Communication at Work, edited by Y. Engeström and D. Middleton, 15–34. Cambridge: Cambridge, UK: Cambridge University Press.
   Google Scholar
• Janssen, C. P., S. F. Donker, D. P. Brumby, and A. L. Kun. 2019. “History and Future of Human-Automation Interaction.” International Journal of Human-Computer Studies 131: 99–107. doi:10.1016/j.ijhcs.2019.05.006.
   Web of Science ®Google Scholar
• Jenkins, D. P., N. A. Stanton, P. M. Salmon, G. H. Walker, and L. Rafferty. 2010. “Using the Decision-Ladder to Add a Formative Element to Naturalistic Decision Making Research.” International Journal of Human-Computer Interaction 26 (2–3): 132–146. doi:10.1080/10447310903498700.
   Web of Science ®Google Scholar
• Ji, Z., N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung. 2022. “Survey of Hallucination in Natural Language Generation.” 55 (12): 1–38. doi:10.48550/arXiv.2202.03629.
   Google Scholar
• Johnson, M., J. M. Bradshaw, and P. J. Feltovich. 2018. “Tomorrow’s Human-Machine Design Tools: From Levels of Automation to Interdependencies.” Journal of Cognitive Engineering and Decision Making 12 (1): 77–82. doi:10.1177/1555343417736462.
   Web of Science ®Google Scholar
• Khurana, D., A. Koli, K. Khatter, and S. Singh. 2023. “Natural Language Processing: State of the Art, Current Trends and Challenges.” Multimedia Tools and Applications 82 (3): 3713–3744. doi:10.1007/s11042-022-13428-4.
   PubMed Web of Science ®Google Scholar
• Klein, G. A. 1989. “Recognition-Primed Decisions.” In Advances in Man-Machine System Research, edited by W. B. Rouse, Vol. 5, 47–92. Greenwich, CT: JAI Press Inc.
   Google Scholar
• Klein, G., D. D. Woods, J. M. Bradshaw, R. R. Hoffman, and P. J. Feltovich. 2004. “Ten Challenges for Making Automation a “Team Player” in Joint Human-Agent Activity.” IEEE Intelligent Systems 19 (06): 91–95. doi:10.1109/MIS.2004.74.
   Web of Science ®Google Scholar
• Klein, K. J., J. C. Ziegert, A. P. Knight, and Y. Xiao. 2006. “Dynamic Delegation: Shared, Hierarchical, and Deindividualized Leadership in Extreme Action Teams.” Administrative Science Quarterly 51 (4): 590–621. doi:10.2189/asqu.51.4.590.
   Web of Science ®Google Scholar
• La Porte, T. R. 1996. “High Reliability Organizations: Unlikely, Demanding and at Risk.” Journal of Contingencies and Crisis Management 4 (2): 60–71. doi:10.1111/j.1468-5973.1996.tb00078.x.
   Google Scholar
• Lake, B., T. Ullman, J. Tenenbaum, and S. Gershman. 2017. “Building Machines That Learn and Think like People.” The Behavioral and Brain Sciences 40: e253. doi:10.1017/S0140525X16001837.
   PubMed Web of Science ®Google Scholar
• Leventi-Peetz, A.-M., T. Ostreich, W. Lennartz, and K. Weber. 2022. “Scope and Sense of Explainability for AI-Systems.” In IntelliSys 2021. Lecture Notes in Networks and Systems, 294. Berlin, Germany: Springer.
   Google Scholar
• Li, Y., and C. M. Burns. 2017. “Modelling Automation with Cognitive Work Analysis to Support Human-Automation Coordination.” Journal of Cognitive Engineering and Decision Making 11 (4): 299–322. doi:10.1177/1555343417709669.
   PubMedGoogle Scholar
• Linde, C. 1988. “Who’s in Charge Here? Cooperative Work and Authority Negotiation in Police Helicopter Missions.” In Proceedings of the Second Annual ACM Conference on Computer Supported Collaborative Work, 52–64. Portland, OR: ACM Press.
   Google Scholar
• Luff, P., and C. Heath. 2000. “The Collaborative Production of Computer Commands in Command and Control.” International Journal of Human-Computer Studies 52 (4): 669–699. doi:10.1006/ijhc.1999.0354.
   Web of Science ®Google Scholar
• Lundberg, J., and A. Rankin. 2014. “Resilience and Vulnerability of Small Flexible Crisis Response Teams: Implications for Training and Preparation.” Cognition, Technology & Work 16 (2): 143–155. doi:10.1007/s10111-013-0253-z.
   Web of Science ®Google Scholar
• Mazaeva, N., and A. M. Bisantz. 2007. “On the Representation of Automation Using a Work Domain Analysis.” Theoretical Issues in Ergonomics Science 8 (6): 509–530. doi:10.1080/14639220600647816.
   Google Scholar
• McKinney, Scott Mayer, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S. Corrado, Ara Darzi, Mozziyar Etemadi, Florencia Garcia-Vicente, Fiona J. Gilbert, Mark Halling-Brown, Demis Hassabis, Sunny Jansen, Alan Karthikesalingam, Christopher J. Kelly, Dominic King, Joseph R. Ledsam, David Melnick, Hormuz Mostofi, Lily Peng, Joshua Jay Reicher, Bernardino Romera-Paredes, Richard Sidebottom, Mustafa Suleyman, Daniel Tse, Kenneth C. Young, Jeffrey De Fauw, and Shravya Shetty. 2020. “International Evaluation of an AI System for Breast Cancer Screening.” Nature 577 (7788): 89–94. doi:10.1038/s41586-019-1799-6.
   PubMed Web of Science ®Google Scholar
• Moy, G., S. Shekh, M. Oxenham, and S. Ellis-Steinborner. 2020. Recent Advances in AI and Their Impact on Defence. DST-Group-TR-3716. Canberra: Defence Science and Technology Group.
   Google Scholar
• Naikar, N. 2013. Work Domain Analysis: Concepts, Guidelines and Cases. Boca Raton, FL: CRC Press.
   Google Scholar
• Naikar, N. 2017. “Cognitive Work Analysis: An Influential Legacy Extending beyond Human Factors and Engineering.” Applied Ergonomics 59 (Pt B): 528–540. doi:10.1016/j.apergo.2016.06.001.
   PubMed Web of Science ®Google Scholar
• Naikar, N. 2018. “Human-Automation Interaction in Self-Organizing Sociotechnical Systems.” Journal of Cognitive Engineering and Decision Making 12 (1): 62–66. doi:10.1177/1555343417731223.
   Web of Science ®Google Scholar
• Naikar, N. 2020. “Distributed Cognition in Self-Organizing Teams.” In Contemporary Research: Models, Methodologies, and Measures in Distributed Team Cognition, edited by M. McNeese, E. Salas, and M. Endsley. Boca Raton, FL: CRC Press.
   Google Scholar
• Naikar, N., and B. Elix. 2016. “Integrated System Design: Promoting the Capacity of Sociotechnical Systems for Adaptation through Extensions of Cognitive Work Analysis.” Frontiers in Psychology 7 (962): 962. doi:10.3389/fpsyg.2016.00962.
   PubMed Web of Science ®Google Scholar
• Naikar, N., and B. Elix. 2021. “Designing for Self-Organisation in Sociotechnical Systems: Resilience Engineering, Cognitive Work Analysis, and the Diagram of Work Organisation Possibilities.” Cognition, Technology & Work 23 (1): 23–37. doi:10.1007/s10111-019-00595-y.
   Web of Science ®Google Scholar
• Naikar, N., G. Moy, H. Kwok, and A. Brady. 2021. “Designing for “Agility” in Envisioned Worlds: Concepts for Collaborative Intelligence in Human-Machine Teams.” In Naturalistic Decision Making and Resilience Engineering Symposium. Toulouse: Naturalistic Decision Making, Resilience Engineering Association, and Fondation pour une culture de sécurité industrielle.
   Google Scholar
• Naikar, N., and A. Saunders. 2003. “Crossing the Boundaries of Safe Operation: An Approach for Training Technical Skills in Error Management.” Cognition, Technology & Work 5 (3): 171–180. doi:10.1007/s10111-003-0125-z.
   Google Scholar
• Naikar, N., A. Moylan, and B. Pearce. 2006. “Analysing Activity in Complex Systems with Cognitive Work Analysis: Concepts, Guidelines and Case Study for Control Task Analysis.” Theoretical Issues in Ergonomics Science 7 (4): 371–394. doi:10.1080/14639220500098821.
   Google Scholar
• Naikar, N., B. Pearce, D. Drumm, and P. M. Sanderson. 2003. “Designing Teams for First-of-a-Kind, Complex Systems Using the Initial Phases of Cognitive Work Analysis: Case Study.” Human Factors 45 (2): 202–217. doi:10.1518/hfes.45.2.202.27236.
   PubMed Web of Science ®Google Scholar
• Naweed, A., and R. Bye. 2008. “Joint Cognitive Systems: Patterns in Cognitive Systems Engineering, Book Review.” Ergonomics 51 (5): 768–770. doi:10.1080/00140130701223774.
   Web of Science ®Google Scholar
• NASEM (National Academies of Sciences, Engineering, and Medicine). 2022. Human-AI Teaming: State-of-the-Art and Research Needs. Washington, DC: The National Academies Press.
   Google Scholar
• NeurIPS 2022 Workshop on Distribution Shifts - Connecting Methods and Applications. 2022. Accessed June 3 2023. https://sites.google.com/view/distshift2022/
   Google Scholar
• Norros, L., and L. Salo. 2009. “Design of Joint Systems: A Theoretical Challenge for Cognitive Systems Engineering.” Cognition, Technology & Work 11 (1): 43–56. doi:10.1007/s10111-008-0122-3.
   Google Scholar
• O’Neill, T. A., N. J. McNeese, A. Barron, and B. Schelble. 2022. “Human-Autonomy Teaming: A Review and Analysis of the Empirical Literature.” Human Factors 64 (5): 904–938. doi:10.1177/0018720820960865.
   PubMed Web of Science ®Google Scholar
• OpenAI. 2023. “GPT-4 Technical Report.” ArXiv.2303.08774. doi:10.48550/arXiv.2303.08774.
   Google Scholar
• Parasuraman, R., M. Mouloua, and R. Molloy. 1996. “Effects of Adaptive Task Allocation on Monitoring of Automated Systems.” Human Factors 38 (4): 665–679. doi:10.1518/001872096778827279.
   PubMed Web of Science ®Google Scholar
• Parasuraman, R., T. B. Sheridan, and C. D. Wickens. 2000. “A Model for Types and Levels of Human Interaction with Automation.” IEEE Transactions on Systems, Man, and Cybernetics. Part A, Systems and Humans: a Publication of the IEEE Systems, Man, and Cybernetics Society 30 (3): 286–297. doi:10.1109/3468.844354.
   PubMed Web of Science ®Google Scholar
• Perrow, C. 1984. Normal Accidents: Living with High Risk Technologies. New York: Basic Books.
   Google Scholar
• Perry, M. 2003. “Distributed Cognition.” In HCI Models, Theories, and Frameworks: Toward a Multidisciplinary Science, edited by J. M. Carroll, 193–223. San Francisco: Morgan Kaufmann.
   Google Scholar
• Phillips, R. O., H. Baharmand, N. Vandaele, C. Decouttere, and L. Boey. 2023. “How Can Authorities Support Distributed Improvisation During Major Crises? A Study of Decision Bottlenecks Arising during Local COVID-19 Vaccine Roll-out.” Journal of Cognitive Engineering and Decision Making 17 (2): 166–187. doi:10.1177/15553434221125092.
   PubMed Web of Science ®Google Scholar
• Pritchett, A. R., S. Y. Kim, and K. M. Feigh. 2014. “Modeling Human-Automation Function Allocation.” Journal of Cognitive Engineering and Decision Making 8 (1): 33–51. doi:10.1177/1555343413490944.
   Google Scholar
• Rasmussen, J. 1969. Man-Machine Communication in the Light of Accident Records (Report No. S-1-69). Roskilde, Denmark: Danish Atomic Energy Commission, Resarch Establishment Risø.
   Google Scholar
• Rasmussen, J. 1983. “Skills, Rules, and Knowledge: Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models.” IEEE Transactions on Systems, Man, and Cybernetics SMC-13 (3): 257–266. doi:10.1109/TSMC.1983.6313160.
   Web of Science ®Google Scholar
• Rasmussen, J. 1986. A Cognitive Engineering Approach to the Modelling of Decision Making and Its Organization (Report No. Risø-M-2589). Roskilde, Denmark: Risø National Laboratory.
   Google Scholar
• Rasmussen, J. 1997. “Merging Paradigms: Decision Making, Management, and Cognitive Control.” In Decision Making Under Stress: Emerging Themes and Applications, edited by R. Flin, E. Salas, M. Strub, and L. Martin, 67–81. Aldershot: Ashgate.
   Google Scholar
• Rasmussen, J., A. M. Pejtersen, and L. P. Goodstein. 1994. Cognitive Systems Engineering. New York: John Wiley & Sons.
   Google Scholar
• Righi, A. W., T. A. Saurin, and P. Wachs. 2015. “A Systematic Literature Review of Resilience Engineering: research Areas and a Research Agenda Proposal.” Reliability Engineering & System Safety 141: 142–152. doi:10.1016/j.ress.2015.03.007.
   Web of Science ®Google Scholar
• Rochlin, G. 1993. “Defining ‘High Reliability’ Organizations in Practice: A Taxonomic Prologue.” In New Challenges to Understanding Organizations, edited by K. H. Roberts, 11–32. New York: Macmillan.
   Google Scholar
• Rochlin, G. I., T. R. La Porte, and K. H. Roberts. 1987. “The Self-Designing High-Reliability Organization: Aircraft Carrier Operations at Sea.” Naval War College Review 40 (4): 76–90.
   Google Scholar
• Roth, E. M., E. P. De Pass, R. Scott, R. Truxler, S. F. Smith, and J. L. Wampler. 2018. “Designing Collaborative Planning Systems: Putting Joint Cognitive Systems Principles to Practice.” In Cognitive Systems Engineering: The Future for a Changing World, edited by P. J. Smith and R. R. Hoffman, 247–268. Boca Raton, FL: Taylor & Francis.
   Google Scholar
• Roth, E. M., C. Sushereba, L. G. Militello, J. Diiulio, and K. Ernst. 2019. “Function Allocation Considerations in the Era of Human Autonomy Teaming.” Journal of Cognitive Engineering and Decision Making 13 (4): 199–220. doi:10.1177/1555343419878038.
   Web of Science ®Google Scholar
• Sadler, M., and N. Regan. 2019. Game Changer: Alphazero’s Groundbreaking Chess Strategies and the Promise of AI. Alkmaar, Netherlands: New in Chess.
   Google Scholar
• Salmon, Paul M., Chris Baber, Catherine Burns, Tony Carden, Nancy Cooke, Missy Cummings, Peter Hancock, Scott McLean, Gemma J. M. Read, and Neville A. Stanton. 2023. “Managing the Risks of Artificial General Intelligence: A Human Factors and Ergonomics Perspective.” Human Factors and Ergonomics in Manufacturing & Service Industries 33 (5): 366–378. doi:10.1002/hfm.20996.
   Web of Science ®Google Scholar
• Shao, Z., R. Zhao, S. Yuan, M. Ding, and Y. Wang. 2022. “Tracing the Evolution of AI in the past Decade and Forecasting the Emerging Trends.” Expert Systems with Applications 209: 118221. doi:10.1016/j.eswa.2022.118221.
   Web of Science ®Google Scholar
• Sheridan, T. B., and W. L. Verplank. 1978. Human and Computer Control of Undersea Teleoperators. Cambridge, MA: MIT Man-Machines System Laboratory.
   Google Scholar
• Shneiderman, B. 2022. Human-Centered AI. Oxford: Oxford University Press.
   Google Scholar
• Silver, David, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. 2017. “Mastering the Game of Go without Human Knowledge.” Nature 550 (7676): 354–359. doi:10.1038/nature24270.
   PubMed Web of Science ®Google Scholar
• Snowden, D. J., and M. E. Boone. 2007. “A Leader’s Framework for Decision Making.” Harvard Business Review 85 (11): 68–76, 149.
   PubMed Web of Science ®Google Scholar
• Suchman, L. A. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge University Press.
   Google Scholar
• van de Ven, G., T. Tuytelaars, and A. Tolias. 2022. “Three Types of Incremental Learning.” Nature Machine Intelligence 4 (12): 1185–1197. doi:10.1038/s42256-022-00568-3.
   PubMedGoogle Scholar
• Vicente, K. J. 1999. Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work. Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Vicente, K. J. 2002. “Ecological Interface Design: Progress and Challenges.” Human Factors 44 (1): 62–78. doi:10.1518/0018720024494829.
   PubMed Web of Science ®Google Scholar
• Vinyals, Oriol, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. 2019. “Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning.” Nature 575 (7782): 350–354. doi:10.1038/s41586-019-1724-z.
   PubMed Web of Science ®Google Scholar
• Weick, K., and K. M. Sutcliffe. 2001. Managing the Unexpected: Assuring High Performance in an Age of Complexity. San Francisco: Jossey Bass.
   Google Scholar
• Wong, W. B. L., P. J. Sallis, and D. O. O’Hare. 1998. “The Ecological Approach to Interface Design: Applying the Abstraction Hierarchy to Intentional Domains.” In Proceedings 1998 Australasian Computer Human Interaction Conference, Los Alamitos, CA, 144–151.
   Google Scholar
• Woods, D. D., J. Tittle, M. Feil, and A. Roesler. 2004. “Envisioning Human-Robot Coordination in Future Operations.” IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) 34 (2): 210–218. doi:10.1109/TSMCC.2004.826272.
   PubMed Web of Science ®Google Scholar
• Woods, D. D., and E. Hollnagel. 2006. Joint Cognitive Systems: Patterns in Cognitive Systems Engineering. Boca Raton, FL: CRC Press.
   Google Scholar
• Zhao, W. X. 2023. “A Survey of Large Language Models.” arXiv prepreprint arXiv:2303.18223.
   Google Scholar