A review of Automatic end-to-end De-Identification: Is High Accuracy the Only Metric?

Figures & data

Figure 1. Example of an end-to-end de-identification process.

Table 1. De-identification systems summary. Machine learning indicates systems that uses machine learning techniques only. Hybrid systems indicates systems that used a combination of machine learning techniques and hand crafted rules.

Architecture	De-identification system
Machine learning	S1 (Zhao et al. (2018)),
	S2 (Chen, Cullen, and Godwin (2015))
	S3 (Dernoncourt et al. (2017)),
	S4 (Yadav et al. (2017)),
	S5 (Lee, Dernoncourt, and Szolovits (2018)),
	S6 (Dernoncourt, Lee, and Szolovits (2017))
Hybrid	S7 (Yang and Garibaldi (2015))
	S8 (Liu et al. (2017))
	S9 (Lee et al. (2016))
	S10 (Dehghan et al. (2015))
	S11 (Yang and Garibaldi (2015))
	S12 (He et al. (2015))
	S13 (Liu et al. (2015))
	S14 (Phuong and Chau (2016))
	S15 (Bui, Wyatt, and Cimino (2017a))
	S16 (Jiang et al. (2017))
	S17 (Lee et al. (2017))
	S18 (Shweta et al. (2016))

Table 2. De-identification systems with overall F-measure $\ge$ 0.95.

	F-measure
De-identification System	i2b2 2006	i2b2 2014	i2b2 2016	MIMIC III
S1	0.9879	0.9805
S3		0.9785		0.9923
S4		0.9746
S5		0.9800	0.9600
S6		0.9770
S7		0.9573
S8		0.9511
S9				0.9926

Table 3. De-identification systems with overall F-measure $\ge$ 0.95.

De-identification	Techniques
System
S1	Recurrent neural network (RNN) + statistical text skeleton approach.
S3	RNN
S4	Conditional Random Field (CRF)
S5	Transfer Learning
S6	Artificial neural networks (ANNs)
S7	CRF + Rule based + Dictionary based
S8	CRF + Rule based
S9	Long Short Term Memories (LSTMs) + human-engineered features

Table 4. F-measure $\ge$ 0.95 for HIPAA categories for de-identification. On occasions where F-measure was not $\ge$ 0.95, the highest score is presented. CONTACT: URL and IP address; ID: BioID, Healthplan, Social Security no, and Vehicle license plate no; Face photo; and Any other unique code are PHIs that were not present in any of the dataset, hence not included here.

PHI categories	Sub-categories	F-measure	Reference	Dataset
(HIPAA)
DATE	Date	$\ge$ 0.95	S2, S3, S4, S8, S10	i2b2 2014
			S11, S12, S13, S18
		$\ge$ 0.95	S3, S9	MIMIC
		$\ge$ 0.95	S14	i2b2 2006
		$\ge$ 0.95	S15, S16, S17, S8	i2b2 2016
NAME	All names	$\ge$ 0.95	S3, S18	i2b2 2014
		$\ge$ 0.95	S3	MIMIC
AGE	Age	$\ge$ 0.95	S1, S3, S8, S18	i2b2 2014
		$\ge$ 0.95	S3	MIMIC
		$\ge$ 0.95	S14	i2b2 2006
		$\ge$ 0.95	S15, S16, S17, S8	i2b2 2016
CONTACT	Phone	$\ge$ 0.95	S3, S11	i2b2 2014
		$\ge$ 0.95	S9	MIMIC
		$\ge$ 0.95	S14	i2b2 2006
		$\ge$ 0.95	S17	i2b2 2016
	Fax	0.80	S2	i2b2 2014
	E-Mail	$\ge$ 0.95	S2, S10, S11	i2b2 2014
ID	Medicalrecords	$\ge$ 0.95	S11, S12	i2b2 2014
		$\ge$ 0.95	S9	MIMIC
	IDNUM	$\ge$ 0.95	S3	i2b2 2014
	Device	0.80	S2, S3	i2b2 2014
	License	$\ge$ 0.95	S17	i2b2 2016
LOCATION	all	$\ge$ 0.95	S3	i2b2 2014
		$\ge$ 0.95	S9	MIMIC

Table 5. Techniques used for the F-measures presented in Table 4 for HIPAA PHI categories.

PHI categories	Sub-categories	Techniques
(HIPAA)
DATE	Date	CRF + Rules + Dictionary (S12); Bi-LSTM (S7);
		CRF + Rules + Keywords (S11);
		Hidden Markov model (HMM-DP) (S2);
		CRF + Rules (S10, S14, S15, S17); LSTM (S16);
		RNN (S3, S18); LSTM + Rules (S9); CRF (S4)
NAME	All names	RNN (S3, S18)
AGE	Age	Bi-LSTM (S7); CRF + Rules (S14, S15, S17);
		LSTM (S16); RNN (S1, S3, S18)
CONTACT	Phone	CRF + Rules + Keywords (S11); RNN (S3);
		CRF + Rules (S14, S17); LSTM + Rules (S9)
	Fax	HMM-DP (S2)
	E-Mail	Rules (S10, S11); HMM-DP (S2)
ID	Medicalrecords	CRF + Rules + Keywords (S11); LSTM + Rules (S9)
	IDNUM	RNN (S3)
	Device	HMM-DP (S2); RNN (S3)
	License	Rules (S17)
LOCATION	all	RNN (S3); LSTM + Rules (S9)

Table 6. The best F-measure for i2b2 extra categories for de-identification. This table includes categories not included in Table 4, but were introduced by i2b2 competitions as additional categories (Stubbs et al. (2015); Stubbs and Uzuner (2015a)). It also provides the techniques used to achieve these F-measures.

PHI categories (i2b2 extra)	Sub-categories	F-measure	Reference	Techniques	Dataset
NAME	Doctor	$\ge$ 0.95	S3, S4	RNN; CRF	i2b2 2014
		$\ge$ 0.95	S2	LSTM + Rules	MIMIC
		$\ge$ 0.95	S14	CRF + Rules	i2b2 2006
		$\ge$ 0.95	S17	CRF + Rules	i2b2 2016
	Patient	$\ge$ 0.95	S3, S4	RNN; CRF	i2b2 2014
		$\ge$ 0.95	S2	LSTM + Rules	MIMIC
		$\ge$ 0.95	S14	CRF + Rules	i2b2 2006
	Username	$\ge$ 0.95	S10	Rules;	i2b2 2014
		$\ge$ 0.95	S11	CRF + Rules + KW;	i2b2 2014
		$\ge$ 0.95	S12	CRF + Rules + Dic	i2b2 2014
LOCATION	Hospital	$\ge$ 0.95	S3	RNN	i2b2 2014
		$\ge$ 0.95	S2	LSTM + Rules	MIMIC
	City	$\ge$ 0.95	S3	RNN	i2b2 2014
	State	$\ge$ 0.95	S3	RNN	i2b2 2014
		$\ge$ 0.95	S2	LSTM + Rules	MIMIC
	Street	$\ge$ 0.95	S3, S9	RNN; HMM-DP	i2b2 2014
		$\ge$ 0.95	S11	CRF + Rules + KW;	i2b2 2014
	Zip	$\ge$ 0.95	S10	Rules	i2b2 2014
		$\ge$ 0.95	S11	CRF + Rules + KW;	i2b2 2014
		$\ge$ 0.95	S12	CRF + Rules + Dic	i2b2 2014
		$\ge$ 0.95	S2	LSTM + Rules	MIMIC
		$\ge$ 0.95	S16	LSTM	i2b2 2016
	Organization	0.82	S3	RNN	i2b2 2014
	Country	$\ge$ 0.90	S3	RNN	i2b2 2014
		$\ge$ 0.90	S2	LSTM + Rules	MIMIC
	Location-Others	0.57	S3	RNN	i2b2 2014
PROFESSION	Profession	0.84	S3	RNN	i2b2 2014

Table 7. Common practices used in surrogate generation and replacement of PHI as outlined in (Johnson et al. (2016); Pantazos, Lauesen, and Lippert (2017); Stubbs and Uzuner (2015b); Stubbs et al. (2015b)).

PHI	Surrogate generation techniques	Manual check
DATE and AGE	Option 1: date shifting where all elements of dates (i.e. day, month and year) are shifted	Yes
	forward by the same random number.
	Option 2: distorted identifier table is used where
	date, month were changed but year was kept the same.
PHONE, FAX,	Randomly created surrogates.	–
URLs, ID
E-MAIL address	Manual replacement	Yes
NAME	Option 1: permutation tables with existing	–
	identifiers are mapped to new ones with
	similar frequency of occurrence.
	Option 2: Mapping between letters, maintaining
	name type and sex.
LOCATION	Random selection of surrogates from pre-compiled	Yes
	list or permutation tables ensuring the type of
	location is matched.
PROFESSION	hand-crafted rules to select from pre-compiled list.	Yes

Zhao, Y.-S., K.-L. Zhang, H.-C. Ma, and K. Li. 2018. Leveraging text skeleton for de-identification of electronic medical records. BMC Medical Informatics and Decision Making 18 (1):18. doi:10.1186/s12911-018-0598-6.

 PubMedGoogle Scholar

Chen, T., R. M. Cullen, and M. Godwin. 2015. Hidden Markov model using Dirichlet process for de-identification. Journal of Biomedical Informatics 58:S60–S66. doi:10.1016/j.jbi.2015.09.004.

 PubMedGoogle Scholar

Dernoncourt, F., J. Y. Lee, O. Uzuner, and P. Szolovits. 2017. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association 24 (3):596–606. doi:10.1093/jamia/ocw156.

 PubMed Web of Science ®Google Scholar

Yadav, S., A. Ekbal, S. Saha, P. S. Pathak, and P. Bhattacharyya. 2017. Patient data de-identification: A conditional random-field-based supervised approach. In Snehanshu Saha, Abhyuday Mandal, Anand Narasimhamurthy, V. Sarasvathi, Shivappa Sangam (Eds.), Handbook of research on applied cybernetics and systems science, 234–53. India: IGI Global.

 Google Scholar

Lee, J. Y., F. Dernoncourt, and P. Szolovits. 2018. Transfer learning for named-entity recognition with neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

 Google Scholar

Dernoncourt, F., J. Y. Lee, and P. Szolovits. 2017. NeuroNER: An easy-to-use program for named-entity recognition based on neural networks. CoRR abs/1705.05487.

 Google Scholar

Yang, H., and J. M. Garibaldi. 2015. Automatic detection of protected health information from clinic narratives. Journal of Biomedical Informatics 58 (Supplement):S30– S38. doi:10.1016/j.jbi.2015.06.015.

 PubMedGoogle Scholar

Liu, Z., B. Tang, X. Wang, and Q. Chen. 2017. De-identification of clinical notes via recurrent neural network and conditional random field. Journal of Biomedical Informatics 75:S34–S42. doi:10.1016/j.jbi.2017.05.023.

 Google Scholar

Lee, J. Y., F. Dernoncourt, O. Uzuner, and P. Szolovits. 2016. Feature-augmented neural networks for patient note de-identification. Proceedings of the Clinical Natural Language Processing Workshop (ClinicalNLP), Japan, 17–22.

 Google Scholar

Dehghan, A., A. Kovacevic, G. Karystianis, J. A. Keane, and G. Nenadic. 2015. Combining knowledge- and data-driven methods for de-identification of clinical narratives. Journal of Biomedical Informatics 58 (Supplement):S53– S59. doi:10.1016/j.jbi.2015.06.029.

 PubMedGoogle Scholar

He, B., Y. Guan, J. Cheng, K. Cen, and W. Hua. 2015. CRFs based de-identification of medical records. Journal of Biomedical Informatics 58 (Supplement):S39– S46. doi:10.1016/j.jbi.2015.08.012.

 PubMedGoogle Scholar

Liu, Z., Y. Chen, B. Tang, X. Wang, Q. Chen, H. Li, J. Wang, Q. Deng, and S. Zhu. 2015. Automatic de-identification of electronic medical records using token-level and character-level conditional random fields. Journal of Biomedical Informatics 58 (Supplement):S47–S52. doi:10.1016/j.jbi.2015.06.009.

 PubMedGoogle Scholar

Phuong, N. D., and V. T. N. Chau. 2016. Automatic de-identification of medical records with a multilevel hybrid semi-supervised learning approach. 2016 IEEE RIVF International Conference on Computing & Communication Technologies, Research, Innovation, and Vision for the Future (RIVF), Vietnam, 43–48.

 Google Scholar

Bui, D. D. A., M. Wyatt, and J. J. Cimino. 2017a. The UAB informatics institute and 2016 CEGS N-GRID de-identification shared task challenge. Journal of Biomedical Informatics 75:S54–S61. doi:10.1016/j.jbi.2017.05.001.

 Google Scholar

Jiang, Z., C. Zhao, B. He, Y. Guan, and J. Jiang. 2017. De-identification of medical records using conditional random fields and long short-term memory networks. Journal of Biomedical Informatics 75:S43–S53. doi:10.1016/j.jbi.2017.10.003.

 Google Scholar

Lee, H.-J., Y. Wu, Y. Zhang, J. Xu, H. Xu, and K. Roberts. 2017. A hybrid approach to automatic de-identification of psychiatric notes. Journal of Biomedical Informatics 75:S19–S27. doi:10.1016/j.jbi.2017.06.006.

 Google Scholar

Shweta, A. Kumar, A. Ekbal, S. Saha, and P. Bhattacharyya. 2016. A recurrent neural network architecture for de-identifying clinical records. Proceedings of the 13th Intl. Conference on Natural Language Processing, Varanasi, India, 188–97.

 Google Scholar

Stubbs, A., C. Kotfila, H. Xu, and Ö. Uzuner. 2015. Identifying risk factors for heart disease over time: Overview of 2014 i2b2/UTHealth shared task track 2. Journal of Biomedical Informatics 58 (Supplement):S67– S77. doi:10.1016/j.jbi.2015.07.001.

 PubMedGoogle Scholar

Stubbs, A., and Ö. Uzuner. 2015a. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics 58 (Supplement):S20– S29. doi:10.1016/j.jbi.2015.07.020.

 PubMedGoogle Scholar

Johnson, A. E., T. J. Pollard, L. Shen, H. L. Li-wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data 3:160035. doi:10.1038/sdata.2016.35.

 PubMed Web of Science ®Google Scholar

Pantazos, K., S. Lauesen, and S. Lippert. 2017. Preserving medical correctness, readability and consistency in de-identified health records. Health Informatics Journal 23 (4):291–303. doi:10.1177/1460458216647760.

 PubMed Web of Science ®Google Scholar

Stubbs, A., and Ö. Uzuner. 2015b. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics 58:S20–S29. doi:10.1016/j.jbi.2015.07.020.

 PubMedGoogle Scholar

Stubbs, A., Ö. Uzuner, C. Kotfila, I. Goldstein, and P. Szolovits. 2015b. Challenges in synthesizing surrogate PHI in narrative EMRs. In Aris Gkoulalas-Divanis and Grigorios Loukides (Eds.) Medical data privacy handbook, 717–35. Switzerland: Springer.

 Google Scholar