AUDITING WITH AI: A THEORETICAL FRAMEWORK FOR APPLYING MACHINE LEARNING ACROSS THE INTERNAL AUDIT LIFECYCLE

Figures & data

Figure 1. Next generation internal audit model.

Source: https://www.cpajournal.com/2020/02/18/the-next-generation-of-internal-audit/

Figure 2. Risk-based internal audit lifecycle.

Source: https://www.iia.org.uk/resources/risk-management/risk-based-internal-auditing/doing-a-risk-based-audit/

Table

#	Sources	Rationale
1	Google Scholar	Preliminary research for scholarly literature, as Google Scholar provides a wide multidisciplinary range of scholarly literature.
2	SSRN	Subject-specific research of scholarly literature published in the social sciences, including humanities and computer science.
3	IEEE Xplore	Research database for scientific and technical content published by the Institute of Electrical and Electronics Engineers – selected as a secondary source of subject-specific research.

Table

#	Stage	ML Technique	Rationale
1	Risk Assessment	Supervised Machine Learning.	During the risk assessment process, the IAF identifies and assesses both the likelihood and potential impact of various risks to the organization (Western Illinois University, n.d.). The IAF then seeks to measure the residual risk of each organizational unit (Arzhenovskiy et al., 2019; Tysiac, 2022) by disaggregating this variable across inherent risk and control effectiveness components.
			Therefore, when framed as a supervised ML problem, we could view the identification of inherent risk and control effectiveness levels as the “target” variable, with the auditor’s chosen risk factors^1 serving as independent variables or “features”. The creation of a supervised machine learning technique should be considered by IAFs to supplement the auditor’s professional judgement with a data-driven, machine learning outcome - which would add potentially significant efficiencies and objectivity.
2	Planning	Generative AI, such as Large Language Models (LLMs). Unsupervised Machine Learning.	The planning stage of an audit typically requires the articulation of key risks and controls for a given audit scope area.
			To this end, LLMs can introduce significant efficiencies^2 (Mökander et al., 2023) by generating first drafts of an audit planning document for an auditor’s independent review and update. Additionally, given that LLMs conceptually draw on an extremely large and complex, generic dataset using complex neural networks (Emett et al., 2023; Mökander et al., 2023), the generated drafts could offer a basic level of objectivity and challenge auditor confirmation bias. In addition to LLMs, the IAF could consider the use of anomaly detection techniques, using unsupervised machine learning capabilities, for key target datasets across the audit scope areas that require exploratory analysis.
3	Fieldwork	Advanced descriptive analytics, with limited applicability for supervised machine learning.	As has been postulated by existing research (Chatzigiannis et al., 2021; Warren & Smith, 2006; Wellmeyer, 2022), regulatory expectations in the field of internal audit necessitate “interpretability” of assurance techniques, whilst generating value. To meet this objective, descriptive analytics, focused on full population testing, should be considered for application at the individual control level. As shown by Bäßler and Eulerich (2023), the use of supervised machine learning is achievable for audit fieldwork; however, lacks sufficient interpretability to enable holistic adoption. On the other hand, descriptive analytics that engages the use of modern analytics (colloquially, “low code”) software enables audit coverage across the entirety of the population, facilitating value generation alongside interpretability.
4	Reporting	Generative AI, such as Large Language Models (LLMs).	LLMs can be used to develop a first draft of an audit report (Emett et al., 2023; Mökander et al., 2023), including key components such as conclusions and draft audit findings, which can introduce significant efficiencies in the delivery of the final internal audit deliverable, namely, the internal audit report.

1. A crucial pre-requisite to enabling machine learning-based capabilities to support audit risk assessment is the identification of the appropriate set of variables, categorical and/or continuous, which could be effectively used to measure inherent risk and/or control effectiveness. These are referred to as “factors” in this paper. Examples of factors could include the number of open audit findings previously raised by the IAF in a given area of review, as this could be used as an indicator of previous control weaknesses identified. Previous research (Bertomeu et al., 2019; Imoniana & Gartner, 2008) highlights the subjective nature of performing this exercise, which allows room for each IAF to determine the most appropriate factors in building a machine learning model for risk assessment.

2. IAFs should be cognizant of inaccurate results generated by LLMs, and therefore, the use of LLMs should be primarily to drive efficiencies in the completion of an audit process within the lifecycle. LLM-generated content should always be reviewed by IAFs for factual accuracy and appropriateness prior to finalization of the relevant audit deliverables.

Western Illinois University. (n.d.). Internal auditing > value added audit services > risk assessment process. Retrieved October 5, 2023, from https://www.wiu.edu/internal_auditing/value_added_audit_services/risk.php#:~:text=During%20the%20risk%20assessment%20process,risk%20is%20at%20manageable%20levels

 Google Scholar

Arzhenovskiy, S. V., Bakhteev, A. V., Sinyavskaya, T. G., & Hahonova, N. N. (2019). Audit risk assessment model. International Journal of Economics & Business Administration, 7(1), 74–85. Retrieved December 20, 2023, from https://doi.org/10.35808/ijeba/253

 Google Scholar

Tysiac, K. (2022). A refreshed focus on risk assessment. Journal of accountancy. Retrieved December 20, 2023, from https://www.journalofaccountancy.com/issues/2022/jan/refreshed-focus-risk-assessment.html

 Google Scholar

Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023). Auditing large language models: A three‑layered approach. SSRN Electronic Journal. Retrieved December 20, 2023, from https://doi.org/10.1007/s43681-023-00289-2

 Google Scholar

Emett, S. A., Eulerich, M., Lipinski, E., Prien, N., & Wood, D. A. (2023). Leveraging ChatGPT for enhancing the internal audit process – A real-world example from a large multinational company. Retrieved August 8, 2023, from https://doi.org/10.2139/ssrn.4514238

 Google Scholar

Chatzigiannis, P., Baldimtsi, F., & Konstantinos, C. (2021, June 21–24). Applied cryptography and network security: 19th international conference, ACNS 2021. Proceedings, Part II (pp. 311–337). Retrieved August 12, 2023, from https://doi.org/10.1007/978-3-030-78375-4_13

 Google Scholar

Warren, J. D., & Smith, M. (2006). Continuous auditing: An effective tool for internal auditors. Internal Auditing, 21(2), 27–35. Retrieved August 12, 2023, from https://ssrn.com/abstract=905144

 Google Scholar

Wellmeyer, P. (2022). In the era of audit data analytics, what’s happened to audit sampling? Retrieved August 12, 2023, from http://dx.doi.org/10.2139/ssrn.3690569

 Google Scholar

Bäßler, T., & Eulerich, M. (2023). Predictive process monitoring for internal audit: Forecasting payment punctuality from the perspective of the three lines model. 6–7, 14–19. Retrieved July 31, 2023, from http://dx.doi.org/10.2139/ssrn.4080238

 Google Scholar

Bertomeu, J., Cheynel, E., Floyd, E., & Pan, W. (2019). Using machine learning to detect misstatements. Review of Accounting Studies. Retrieved August 12, 2023, from https://ssrn.com/abstract=3496297

 Web of Science ®Google Scholar

Imoniana, J. O., & Gartner, I. R. (2008). Towards a multi-criteria approach to corporate auditing risk assessment in Brazilian context. Retrieved August 8, 2023, from https://doi.org/10.2139/ssrn.1095950

 Google Scholar