Optimizing Fraudulent Firm Prediction Using Ensemble Machine Learning: A Case Study of an External Audit

ABSTRACT

This paper is a case study of utilizing machine learning for developing a decision-making system for auditors before initializing the audit fieldwork of public firms. Annual data of 777 firms from 14 different sectors are collected and a MCTOPE (Multi criteria ToPsis based Ensemble) framework is implemented to build an ensemble classifier. MCTOPE framework optimizes the performance of classification during ensemble building using the TOPSIS multi-criteria decision-making algorithm. Ensemble machine learning is used for optimizing the prediction performance of suspicious firm predictor in the previous work available at https://www.tandfonline.com/doi/full/10.1080/08839514.2018.1451032. After achieving an accuracy of 94.6% and AUC (area under the curve) value of 0.98, this ensemble classifier is employed in a web application developed for auditors using Python and R script for the prediction of suspicious firm before planning an external audit. The performance of an ensemble classifier is validated using K-fold cross validation technique and is found to be better than the state-of-the-art classifiers.

Introduction

Fraud is a critical issue worldwide. Firms that resort to the unfair practices without the fear of legal repercussion have a grievous consequences for the economy and individuals in the society. Auditing practices are responsible for the fraud detection. Audit is defined as the process of examining the financial records of any business to corroborate that their financial statements are in compliance with the standard accounting laws and principles (Cosserat 2009). Data analytics tools for an effective fraud management have become the need of the hour for an audit. The possibilities that how data analytics can improve the quality of process is published in Emerging Assurance Technologies Task Force of the AICPA Assurance Services Executive Committee (ASEC) (AICPA Staff 2014).

When the audits are performed by any external audit company, the risk assessment plays a vital role in deciding the amount of fieldwork that would be required before actually visiting the official firms. The complete process of risk assessment during audit is explained in detail in the previous work (Hooda 2018). The prime goal of an auditor during an audit-planning phase is to follow a proper analytical procedure to impartially and appropriately identify the firms that resort to a high risk of unfair practices. Identifying fraudulent firms can be studied as a classification problem. The purpose of classifying the firms during the preliminary stage of an audit is to maximize the field-testing work of high-risk firms that warrant significant investigation.

Many researchers have employed algorithms like artificial neural network, logistic regression, decision trees, and bayesian belief networks for detecting management fraud in the financial statements (Fanning 1998; Green 1997; Spathis 2002). Ensemble machine learning method is also applied successfully for improving the classification accuracies of the auditing task (Kotsiantis 2006). Machine learning algorithms like support vector machine, logistic regression, probabilistic neural network, genetic algorithm, etc. are also combined with feature selection methods in order to prove their usability in detecting fraud in the Chinese firms (Ravisankar 2011). During audit-planning, auditors examine the business of different government offices but target to visit the offices with very-high likelihood and significance of misstatements.

As stated by Wolpert, there is no single best algorithm which is applicable for all the possible cases of problems (Wolpert and Macready 1997). Furthermore, a lot of research efforts have been made for improving the performance of machine learning models by developing an ensemble-classifier which is constructed from diverse machine learning models (Dietterich 2000; Qi 2012). From a practical point of view, multiple-opinions are unfailingly better than a single opinion in any decision-making process. Ensemble learning serves as a powerful tool in machine learning as it employs multiple classifiers and works on optimizing the performance of base classifiers separately. Although it cannot always guarantees a success, but generally, it reduces variance and offers better performance than a single classifier solution (Dietterich 2000; Qi 2012; Zhang and Ma 2012). By choosing a specific aggregation technique like majority voting, boosting, bagging, etc., an ensemble classifier aids to scrutinize the risk of obtaining poor results from a single classifier system.

This research work is a case study of an external government audit company which is also an external auditor of government firms of India. Complex audit data is collected and an ensemble classifier is built using MCTOPE (Multi criteria ToPsis based Ensemble) framework, which implements the Technique for Order of preferences by similarity to Ideal Solution (TOPSIS) Multi-Criteria assessment algorithm (Majid 2012). The performance of the built ensemble is tested using K-fold cross validation technique and is also compared with other state-of-the-art methods. This ensemble is employed in a Fraudulent Firm Prediction Web Application (built in Django Python Framework) for predicting the high-risk firms.

The goal of the research is to help the auditors by building a classification model that can predict the fraudulent firm on the basis of the present and historical risk factors. The validity of proposed framework is tested using the K-fold cross validation method, and then the proposed method is applied to suspicious firm prediction problem.

The rest of this paper is structured as follows. Section 2 describes the audit data, features, and experimental setup. The proposed framework and ensemble model building technique are presented in Section 3. Performance analysis, results and analysis are discussed in Section 4. Finally, Section 5 concludes this paper and points out the scope of further research.

Data and Experimental Setup

Data Collection

Exhaustive 1 year and 6 months non-confidential data firms is collected from the Audit General Office (AGO) of India. There are total 776 firms from 46 different cities of a state that are listed by the auditors for targeting the next field-audit work. The target-offices are listed from 14 different sectors. Detailed description of data and features is presented in the previous paper (Hooda 2018)

Experiment Setting

Ten state-of-the-art classification methods namely decision tree (DT)(Quinlan 1986), adaboost (AB)(Schapire 1999), random forest (RF) (Liaw et al. 2002), support vector machine (SVM) (Keerthi et al. 0000), probit linear model (PLM) (Chambers 1977), neural network (NN)(Russell 2003), decision stump (DSM)(Iba et al. 1992), J48 (Ross Quinlan 1996), Naive Bayes (NB)(Rish 2001), and Bayesian (BN) are employed to make the pool of classifiers called model-pool for ensemble building. The R ‘caret’ package is used to implement the various classification models. The models are available in R open-source software. R is licensed under GNU GPL. The complete description of model’s parameter setting and required packages is summarized in the previous work (Hooda 2018). The purpose is to measure the prediction performance of the model when it is up- and running and then predicting the suspicious firm class of the new samples without the benefit of knowing the true risk-class of the samples.

Prediction Model

The prediction engine automates the process of ensemble building. The complete process is presented in Figure 1.

Figure 1. Architecture of prediction engine in MCTOPE Architecture; M: Machine learning model; RS: Random sample of training dataset.

In the first step, the data is prepared using data preparator. Data preparation works on cleaning, feature extraction and feature selection process. In the second step, random samples are generated to train the randomly selected classifiers in the model pool. The decision file is generated to collect the predictions of the randomly selected classifiers from the model-pool. The sample decision file is presented in Table 1. The sample decision file contains the predictions of six selected classifiers, chosen randomly from the model pool. D represents the ensemble decision by majority voting. In the third step, initially, a preliminary-ensemble is generated by combining m(1 $\le$ m $\le$ 10) different machine learning model’s decision by random using the majority voting aggregation technique. TOPSIS multi-criteria decision score of preliminary-ensemble is evaluated using six different performance measures. In the next iteration, a new ensemble is produced using different training samples, and with a new set of models in model-pool. The performance of this ensemble is compared with the preliminary ensemble, and the model with the higher TOPSIS score is saved. This phase is called optimization for the best model search. At the end n iterations (say n = 5000 for small dataset), a final ensemble with the highest performance score is declared as the Winning-Ensemble.

Topsis Performance Evaluation

Accuracy of the classifier is a popular approach to compare the level of competence among several classifiers (Bradley 1997). Using a single evaluation metric for comparing the performance of machine learning model is error-prone approach. In this work, the proposed ensemble is no longer evaluated only the accuracy of classification, which is quite different from the available techniques. For comprehensive evaluation of ensemble model performance, multiple performance metrics should be considered. Among the several solutions available, Multi-Criteria Decision making is the most prevalent approach. Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) is a popular multi-criteria analysis technique (Triantaphyllou 2000). TOPSIS score does not only calculates the accuracy but also focuses on sensitivity, specificity, and area under the ROC curve (AUC) of the classifier. Choosing the evaluation-criteria that suits the goal of improving the classification performance is an important step of this process. The proposed framework considers multi-criteria performance metrics of confusion matrix (Table 2) presented in Table 3 for testing the performance of suspicious firm classification.

Table 1. Decision file sample.

Iteration	Machine learning models						Decision
	RF	AB	PLM	SVM	NN	NB	D
i	f	t	t	f	t	t	t

Table 2. Confusion matrix.

	True Reference
Predicted Condition	Suspicious	Non Suspicious
Suspicious	True positive X	False Negative Z
Non Suspicious	False Positive Q	True Negative Y

Table 3. Performance evaluation metrics.

Performance Metric	Formula
Type-I error	Q
Type-II error	Z
Sensitivity	$X/(X+Z)$
Specificity	$Y/(Q+Y)$
Accuracy	$(X+Y)/(X+Z+Q+Y)$
F Score	$(2\ast X)/(2\ast X)+(Q+z)$
MCC	$(X\ast Y)-(Q\ast Z)/SQRT((X+Q)+(X+Z)+(Y+Q)+(Y+Z))$

A TOPEES assessment score function (Triantaphyllou 2000) is introduced for comprehensive performance assessment. It works on the principle of finding an alternative closest to the ideal solution and farthest from the negative-ideal solution. Ideal solution i is the set of evaluation-criteria solution with maximum benefit and can be described as

(1) $\begin{array}{c}i=\{max(accuracy),min(error),max(sensitivity),\\ max(specificity),max(MCC),max(Fscore),max(AUC)\}\end{array}$(1)

Negative ideal solution j is the solution with the maximum loss and can be described as

(2) $\begin{array}{c}j=\{min(accuracy),max(error),min(sensitivity),\\ min(specificity),min(MCC),min(Fscore),min(AUC)\}\end{array}$(2)

Experimental Results and Discussion

MCTOPE framework builds an ensemble of naive bayes and decision tree classifiers with the highest performance score for the collected audit data. For testing, the experiments are designed to use 10-fold cross validation method. The data set is divided into 10 equal size subsets. Ensemble algorithm is built to train the nine subset folds and testing on the last subset fold. To test the robustness of designed ensemble classifier, the process is iterated. To evaluate the performance of the proposed framework, different performance parameters namely accuracy, sensitivity, specificity, F measure, MCC and Area under curve (AUC) are used.

The performance of the built ensemble using Naive Bayes (NB) and Decision Tree (DT) as base classifiers is graphically depicted on each fold of 10-cross validation method in Figure 2. It can be observed that the accuracy, sensitivity, and specificity of the built ensemble is better than the base classifiers and it is quite robust as value of accuracy is not changing abruptly. It can be observed that the AUC of the ensemble is better than the base classifiers and it is quite robust as values are stable.

Figure 2. Accuracy, sensitivity, specificity and AUC results using K fold cross validation on audit testing dataset using MCTOPE ensemble classifier.

Comparison with State-of-the-Art Methods

In order to perform the comparison analysis, the state-of-the-art classifiers are explored in the area of auditing and finance. The results of the proposed framework is compared with the outstanding classical classifiers like support vector machine (SVM), random forest, neural network, C4.5, adaboost, naive bayes, etc. The results are presented in Table 4. The best values of the performance metrics are highlighted. The performance of an ensemble is quite better than the other classifiers. If the accuracy parameter is compared, performance of an ensemble classifier is closer to the random forest and adaboost classifiers. This can be due to the fact that random forest and adaboost models are also ensemble-based classifiers. Sensitivity of the built ensemble is closer to the naive bayes classifier. Sensitivity of the built ensemble is closer to the decision tree classifier. If F metric is compared, values of the built ensemble is closer to the random forest classifier. MCC value of built ensemble is lower than random forest, J48, adaboost, bayesian network, decision tree, and PLM classifiers. The AUC value of the built ensemble is the best and quite closer to 1. To check the overall performance of the built ensemble, multi-criteria based TOPSIS performance scores can be compared.

Table 4. Comparison the performance of state-of-the-art classifiers with built ensemble classifier.

Metric	Ensemble	SVM	Rf	J48	NN	Aboost	NB	BN	DT	PLM	DS
Accuracy(%)	94.6	65.63	93	92	79.15	92.65	82	91	90	92	87.68
Sensitivity	1.00	0.50	0.95	0.97	0.89	0.96	0.99	0.95	0.92	0.95	0.98
Specificity	0.92	0.81	0.91	0.89	0.63	0.89	0.65	0.87	0.90	0.88	0.75
F measure	0.94	0.64	0.93	0.93	0.79	0.92	0.81	0.91	0.91	0.92	0.87
MCC	0.87	0.32	0.87	0.86	0.59	0.85	0.69	0.84	0.83	0.84	0.77
AUC	0.98	0.65	0.96	0.93	0.86	0.93	0.92	0.95	0.89	0.93	0.86

For a comprehensive performance check, TOPSIS scores of the base classifiers are compared in Figure 3. It is clear in the figure that the overall performance of output ensemble is exceptionally good. It demonstrates that the MCTOPE framework builds an ensemble classifier, which is better than the other base classifiers on the same audit sample dataset.

Figure 3. TOPSIS performance comparison analysis.

Implementation

An ensemble classifier with the highest performance score is implemented as a Fraudulent Firm Web Application using Python Django Web framework. Figure 4 presents the main page of the web application. A user can submit the values of the features of any firm to classify the firm as Suspicious or Non Suspicious. In order to predict a new firm for next year fraud risk, this web-application takes input of the important features shown in Figure 4 and predicts the probability of risk using ensemble model, working in the back-end.

Figure 4. Fraudulent firm web application.

Conclusion

In this paper, a case study of an external audit company is studied and an ensemble classifier-based web application is built to help in the decision-making process of predicting the suspicious firm before an external audit. Different from the traditional approach of classification, a MCTOPE framework builds an efficient ensemble classifier by optimizing the overall performance of the classifier using TOPSIS algorithm, a multi-criteria decision-making technique. After more than 1000 iterations, the performance of the final ensemble using Naive Bayes and Decision Tree as base classifiers with the highest TOPSIS performance score is depicted on each fold of the 10-fold cross validation method. The classifier is found to be robust with an accuracy, sensitivity, specificity, F measure, MCC, AUC of 94, 1, 0.92, 0.94, 0.83, and 0.98, respectively. When compared with the state-of-the-art classifiers, it is found to be better than the available methods, serving as a proof of eligibility of classifiers to perform an efficient fraudulent firm prediction in the audit fieldwork decision-making process.

For future works, we are targeting to offer the auditors to handle the last 10-year data of firms on the top of advance big data techniques like Hadoop, Spark, etc. This research work is supported by Ministry of Electronics and Information Technology (MEITY), Govt. of India (Grant No. DoRSP/1633). The authors wish to thank the auditors of audit office for their assistance, time, and continued support. The authors are grateful for their helpful feedbacks and comments on early version of this work.

Acknowledgments

The authors wish to thank the auditors of audit office for their assistance, time, and continued support. The authors are grateful for their helpful feedbacks and comments on the early version of this work.

Additional information

Funding

This research work is supported by Ministry of Electronics and Information Technology (MEITY), Govt. of India (Grant No. DoRSP/1633).