Credit Card Fraud Detection with Automated Machine Learning Systems

Figures & data

Table 1. Overview of the amount variable

Panel A: Descriptive statistics
	Mean	Standard deviation	Kurtosis	Skewness	Normality (Jarque-Bera test)	Augmented Dickey-Fuller (ADF) test	Kwiatkowski–Phillips–Schmidt–Shin (KPSS) test
	€88.35	€250.12	848.07	16.98	8,488*	−473.71*	6.68
Panel B: Data distribution according to the value of each transaction.
	>€2,000	(€1,000- €2,000]	(€500- €1,000]	(€300- €500]	(€200- €300]	(€0, €200]
	0.237%	0.795%	2.178%	3.131%	3.784%	89.875%

Note: * denotes rejection of the null hypothesis at the 5% level of significance. The null hypothesis of the Jarque – Bera test is that the data originate from a normal distribution. The null of the ADF test is that the data are nonstationary, while the null of the KPSS test is that the data are stationary.

Table 2. Accuracy metrics

Metrics	Train	Test
Overall Accuracy	0.999 [0.999,1.000]	0.999 [0.999,1.000]
Area Under the ROC Curve	0.973	0.981
Balanced Accuracy for class 0	0.891 [0.872,0.910]	0.924 [0.867, 0.972]
Balanced Accuracy for class 1	0.891 [0.872,0.910]	0.924 [0.867,0.972]
F-measure for class 0	0.999 [0.999,1.000]	0.999 [0.999, 1.000]
F-measure for class 1	0.787 [0.757,0.815]	0.821 [0.727,0.898]
Precision for class 0	0.999 [0.999,1.000]	0.999 [0.999, 1.000]
Precision for class 1	0.791 [0.752,0.828]	0.796 [0.675,0.905]
Recall for class 0	0.999 [0.999,1.000]	0.999 [0.999, 1.000]
Recall for class 1	0.783 [0.744,0.815]	0.848 [0.735,0.944]
Sensitivity for class 0	0.999 [0.999,1.000]	0.999 [0.999, 1.000]
Sensitivity for class 1	0.783 [0.744,0.820]	0.848 [0.735, 0.944]
Specificity for class 0	0.783 [0.744,0.820]	0.848 [0.735, 0.944]
Specificity for class 1	0.999 [0.999,1.000]	0.999 [0.999, 0.999]

Note: Class 0 denotes legitimate transactions, while Class 1 denotes fraudulent ones. 95% confidence intervals are reported in brackets.

Figure 1. The ROC curve for Class 1 (fraudulent transactions), along with the respective 95% confidence intervals. The thick dashed (blue) line is the estimated ROC from the training data. Different points on the ROC curve provide a different trade-off between False Positive Rate (FPR) and the true positive rate (sensitivity) of the classifier. JAD can output models operating at different FPRs by selecting any of the circles.

Table 3. The confusion matrix

	Train		Test (Out-of-sample forecasting)
Panel A: Cases
	Actual legitimate	Actual fraudulent	Actual legitimate	Actual fraudulent
Estimated legitimate	256,014	97	28,199	7
Estimated fraudulent	92	349	10	39
Total	256,106	446	28,209	46
Panel B: Percentages
	Actual legitimate	Actual fraudulent	Actual legitimate	Actual fraudulent
Estimated legitimate	99.96%	21.75%	99.96%	15.22%
Estimated fraudulent	0.04%	78.25%	0.04%	84.78%
Total	100%	100%	100%	100%

Table 4. Descriptive statistics of out-of-sample forecasts

	Fraudulent transactions identified	Fraudulent transactions missed	False Alarms
Minimum	0.77€	1.00€	0.77€
Maximum	1809,68€	311,91€	89,99€
Mean	193,21€	68,23€	10,84€
Sum	7,535.24€	477.64€	97.53€
Observations	39	7	10
Number of transactions below 10€	21	3	9
Number of transactions between 10€ and 50€	1	2	1
Number of transactions between 50€ and 100€	4	0	0
Number of transactions between 100€ and 500€	8	2	0
Number of transactions above 500€	5	0	0

Table 5. Comparison to earlier studies

Metrics/Model	Plakandaras et al. (JAD)	Dal Pozzolo et al. (2015)	Johm, Adetunmbi, and Oluwadare (2017)	Fiore et al. (2019)
Overall Accuracy	0.999		0.969	0.999
Area Under the ROC Curve	0.981	0.975–0.999
Balanced Accuracy	0.924		0.928	0.851
F-measure	0.821		0.561	0.811
Precision	0.796		0.410	0.958
Recall	0.848		0.884	0.958
Sensitivity	0.848		0.884	0.702
Specificity	0.999		0.971	0.999

Dal Pozzolo, A., O. Caelen, A. J. Reid, and G. Bontempi. 2015. Calibrating probability with undersampling for unbalanced classification. Symposium on Computational Intelligence and Data Mining (CIDM) Cape Town, South Africa, IEEE.

 Google Scholar

Johm, A., A. O. Adetunmbi, and S. A. Oluwadare. 2017. Credit card fraud detection using machine learning techniques: A comparative analysis, IEEE International Conference on Computing Networking and Informatics (ICCNI), Lagos, Nigeria.

 Google Scholar

Fiore, U., A. De Santis, F. Perla, P. Zanetti, and F. Palmieri. 2019. Using generative adversarial networks for improving classification effectiveness in credit card fraud detection. Information Sciences 479:448–55. doi:10.1016/j.ins.2017.12.030.

 Web of Science ®Google Scholar