Identifying chronic disease patients using predictive algorithms in pharmacy administrative claims: an application in rheumatoid arthritis

Figures & data

Figure 1. Overview of model development and evaluation. AUROC, area under the receiver operating characteristic curve; MCC, Matthews correlation coefficient; ML, machine learning; RA, rheumatoid arthritis.

Figure 2. F1 scores for regression models and machine learning algorithms across analytic samples. ETN, etanercept; LDA, linear discriminant analysis; QDA, quadratic discriminant analysis; RF, random forest; SVM, support vector machine.

Figure 3. AUROC for regression models and machine learning algorithms across analytic samples. AUROC, area under the receiver operating characteristic curve; ETN, etanercept; LDA, linear discriminant analysis; QDA, quadratic discriminant analysis; RF, random forest; SVM, support vector machine.

Table 1. Predictive performance of statistical functions and machine learning algorithms (all-patients sample).

Model	F1 Score	Accuracy	Precision	Sensitivity	AUROC	TPs	FPs	FPR	FNs	FNR	MCC
Regression models
Logistic regression	83.96	83.79	85.80	82.20	83.85	34,821	5761	14.50	7542	17.80	67.66
Linear regression	83.34	83.17	85.23	81.53	83.23	34,538	5985	15.07	7826	18.47	66.43
Machine learning algorithms
Ridge classifier	83.34	83.19	85.23	81.53	83.23	34,538	5985	15.07	7826	18.47	66.43
Linear SVM	83.73	83.57	85.64	81.90	83.63	34,695	5816	14.64	7669	18.10	67.23
Naïve Bayes classifier	71.73	73.44	79.60	65.28	73.72	27,656	7089	17.85	14,708	34.72	47.98
LDA	83.35	83.19	85.25	81.53	83.24	34,537	5974	15.04	7826	18.47	66.45
QDA	49.07	63.19	85.97 (+0.2%)	34.50	64.14	14,617	2471	6.22 (–57.1%)*	27,747	65.50	34.96
RF	84.55 (+0.7%)	84.05 (+0.3%)	84.56	84.53 (+2.8%)	84.04 (+0.2%)	35,809 (+2.8%)	6536	16.46	6555	15.47 (–13.1%)	68.07 (+0.6%)
Decision trees	77.92	77.37	78.47	77.39	77.37	32,786	8998	22.65	9578	22.61	54.72
Neural network	82.57	82.22	83.75	81.62	82.24	34,579	6810	17.15	7785	18.38	64.62

All the values except TPs, FPs, and FNs were presented as percentages. F1 score ranges from 0 to 1 and results are presented as a percentage (where 100% represents score of 1, 50% represents score of 0.5, and 0% represents a score of 0) for ease of interpretation. AUROC, area under the receiver operating characteristic curve; FPs, false positives; FNs, false negatives; FNR, false negative rate; FPR, false positive rate; LDA, linear discriminant analysis; MCC, Matthews correlation coefficient; QDA, quadratic discriminant analysis; RF, random forest; SVM, support vector machine; TPs, true positives.

Bolded and shaded cells indicate the “best” value for a given performance metric. The FNR for RF was >10% versus logistic regression. *Despite a > 10% difference in FPR versus logistic regression, the F1 score for QDA was much lower than that for logistic regression.

Figure 4. False positive rate (A) and false negative rate (B) for regression models and machine learning algorithms across analytic samples. ETN, etanercept; LDA, linear discriminant analysis; QDA, quadratic discriminant analysis; RF, random forest; SVM, support vector machine.

Table 2. Predictive performance of statistical functions and machine learning algorithms (all-etanercept patients sample).

Model	F1 Score	Accuracy	Precision	Sensitivity	AUROC	TPs	FPs	FPR	FNs	FNR	MCC
Regression models
Logistic regression	84.51	81.45	84.53	84.48	80.71	18,647	3413	23.07	3425	15.52	61.41
Linear regression	84.20	80.94	83.59	84.82	79.98	18,721	3676	24.85	3351	15.18	60.20
Machine learning algorithms
Ridge classifier	84.19	80.94	83.59	84.81	79.98	18,718	3675	24.84	3353	15.19	60.19
Linear SVM	84.28	81.22	84.45	84.12	80.51	18,566	3418	23.10	3506	15.88	60.95
Naïve Bayes classifier	69.08	67.78	81.19	60.11	69.67	13,267	3073	20.77	8805	39.89	38.82
LDA	84.17	80.93	83.68	84.66	80.01	18,686	3644	24.63	3385	15.34	60.21
QDA	43.87	54.22	82.58	30.31	60.10	6690	1497	10.12 (–56.1%)*	15,381	69.69	24.14
RF	84.73 (+0.3%)	81.36	83.15	86.38	80.13	19,064	3863	26.11	3007	13.62	60.92
Decision trees	78.31	74.19	78.80	77.81	73.29	17,175	4620	31.22	4897	22.19	46.45
Neural network	82.90	77.99	77.49	89.13 (+5.5%)	75.25	19,672 (+5.5%)	5716	38.63	2400	10.87 (–30.0%)*	53.46

All the values except TPs, FPs, and FNs were presented as percentages. F1 score ranges from 0 to 1 and results are presented as a percentage (where 100% represents score of 1, 50% represents score of 0.5, and 0% represents a score of 0) for ease of interpretation. AUROC, area under the receiver operating characteristic curve; FPs, false positives; FNs, false negatives; FNR, false negative rate; FPR, false positive rate; LDA, linear discriminant analysis; MCC, Matthews correlation coefficient; QDA, quadratic discriminant analysis; RF, random forest; SVM, support vector machine; TPs, true positives. Bolded and shaded cells indicate the “best” value for a given performance metric. *Despite a > 10% difference in FPR and FNR versus logistic regression, the F1 scores for QDA and neural network were much lower than those for logistic regression.