Artificial intelligence in digital histopathology for predicting patient prognosis and treatment efficacy in breast cancer

Figures & data

Figure 1. The differences between traditional machine learning (ML) and deep learning (DL). AI: artificial intelligence; CNN: convolutional neural network. An input layer, an output layer, and multiple hidden layers make up convolutional networks.

Figure 2. Artificial intelligence (AI) and Digital Pathology Workflow. DIA: data-independent acquisition. Initially, tissue samples are stained to highlight key histological features, followed by the digitization of the samples into high-resolution digital images using scanners. AI-based algorithms are subsequently applied to identify and quantify relevant features (ex-TIL count, stromal, and tumor area) within the images. The final step involves evaluating the association between the identified features and patient survival outcomes through statistical analysis.

Table 1. Digital pathology related artificial intelligence models in oncology.

Biomarkers	Purpose	Author	Cancer Type	Dataset	Model	Performance
Tumour Infiltrating Lymphocytes (TILs)	Prognostic	Bai et al., 2021	TNBC	Discovery set: 171 patients Validation sets: 749 patients in total	Neural Network (CNN11)	High easTILs (≥19.9): HR:0.57, CI: 0.42–0.78, p < 0.0003.
		Heindl et al., 2018	ER+ Breast Cancer	Cohort of 1,178 postmenopausal patients treated with 5 years tamoxifen or anastrozole.	Fully automated H&E-stain image analysis algorithm	Immune Hotspot LR-χ2 = 14.06, p < 0.001_for 0–10 years; LR-χ2 = 6.24, p = 0.01 for 0–5 years; LR-χ2 = 7.89, p = 0.005 for 5–10 years.
		Thagaard et al., 2021	TNBC	Cohort of 257 patients	Multiple CNNs trained with a VGG-based encoder	Overall survival (HR: 0.81 CI: 0.72–0.92 p = 0.001)
	Predictive	Lee et al., 2022	TNBC	Cohort of 954 patients	Lunit SCOPE IO	iTIL score was found to’ultivariable analysis (adjusted odds ratio 1.211 (95% CI 1.125–1.304) per 1 point (%) change in the score, p < 0.001).
		Park et al., 2022	NSCLC	Development: 3,166 WSI of 25 cancer types Validation: lung adenocarcinoma (LUAD, n = 461) and lung squamous cell carcinoma (LUSC, n = 462) images from The Cancer Genome Atlas (TCGA), as well as NSCLC primary tumor tissues from Samsung Medical Center (SMC, n = 1,205) and Seoul National University Bundang Hospital (SNUBH, n = 261)	Lunit SCOPE IO	In the PD-L1 TPS 1%-49% subgroup, the AUROC for prediction of tumor response to ICI by IP was 0.7609, and the AUROC by PD-L1 TPS status was 0.5561 (p < .05).
Mitosis/LDO	Prognostic	Guillaud et al., 2017	Breast	A cohort of 95 patients	Forward stepwise linear discriminant analysis	Logrank test: p < 0.0001
	Predictive	Zhao et al., 2021	Colorectal	A cohort of 496 patients with stage I, II, or, III	GLEM4D	Stage II patients with at least one high-risk factor (non-diploidy, high-stroma, chromatin heterogenous), patients who received adjuvant therapy showed significantly longer OS (72.1% vs 48.3%, p = 0.007) and DFS (64.5% vs 43.9%, p = 0.015)
Nuclear Features	Prognostic	Beck et al., 2011	Breast	Two independent cohorts comprising a total of 576 patients	C-Path	Overall Survival (log-rank: p = 0.001)
	Predictive	Whitney et al., 2018	Breast	A cohort of 178 early-stage ER+ breast cancer	Random Forest, Neural Network, Support Vector Machine, and Linear Discriminant Analysis, via 3-fold cross validation.	Low Oncotype DX (ODx) risk categories vs. High ODx risk categories (AUC = 0.72)
		Lu et al., 2018	Breast	A cohort of 278 early-stage ER+, LN breast cancer patient	Linear discriminant analysis classifier	Multivariate analysis: hazard ratio (95% confident interval) = 3.17(0.33–30.46), p = 0.01039.
		Doddington et al., 2021	HER2+ and TNBC	Fifty-eight HER2+ or TNBC patients	Multiple Deep CNN	Model successfully classified 79% of cases (62% for pPR and 89% for pCR)
		Saednia et al., 2022	Breast	A cohort of 149 patients treated with NAC	weighted U-Net model.	NAC response prediction at pre-treatment: AUC of 0.90, sensitivity of 85%, and a specificity of 82%
Tumor/Stroma Ratio	Prognostic	Yan et al., 2022	Breast	A cohort of 240 patients	Open CV	Low vs. High TSR for predicting DFS: p < 0.0001
		Millar et al., 2020	Luminal and TNBC	A cohort of 403 luminal and 244 TNBC	QuPath	Low TSR for TNBC: OS (HR 1.9, CI 1.1–3.3, p = 0.021); Low TSR for Luminal breast cancer: (HR 0.6, CI 0.4–0.8, p = 0.001)
		Abubakar et al., 2021	Breast	An ethnically diverse group of 2,084 breast cancer patients from China, Poland, and the United States	HALO	SCD was predictive of poor clinical outcomes: multivariable HR (95%CI) fourth(Q4) vs first(Q1) quartile = 1.86(1.06–3.26);Ptrend = 0.03), Poland (HR(95%CI)Q4 vs Q1 = 1.80(1.12–2.89);Ptrend = 0.01), and United States (HR(95%CI)Q4 vs Q1 = 2.42(1.33–4.42);Ptrend = 0.002).
	Predictive	Li et al., 2022	Breast	A multicentre dataset of 1035 patients	OSAMTL-DNS	TSR predicted pCR with an AUC of 0.745, 0.673, and 0725 in the validation cohorts
Deep Learning	Prognostic	Xu et al., 2021	Breast	A cohort of 1,058 breast cancer patients	attention-based multiple instance-learning (AMIL) framework	Predicting ALN metastasis: AUC of 0.816 (95% confidence interval (CI): 0.758, 0.865)
		Phan et al., 2021	Breast	A cohort of 138 patients	VGG16, ResNet50, ResNet101, Inception_ResNet, EfficientB5, and Xception	Accuracy of models for predicting recurrence: 0.84, 0.85, 0.83, 0.84, 0.87, and 0.91, respectively.
		Yang et al., 2022	HER2+ Breast Cancer	A cohort of 127 HER2+ breast cancer patients	Deep CNN	AUC of 0.76 for predicting recurrence and metastasis risk
	Predictive	Li et al., 2021	Breast	A cohort of 540 breast cancer patients receiving NAC	CNN	AUC of 0.847 in predicting pCR directly, and achieved accuracy, F1 score, and AUC of 0.853, 0.503, and 0.822
		Farahmand et al., 2022	HER2+ Breast Cancer	A cohort of 187 HER2+ breast cancer patients who received neoadjuvant targeted therapy with trastuzumab± pertuzumab prior to definitive surgery was used for training.	Inception v3 architecture	AUC of 0.80 for predicting trastuzumab response
		Lou et al., 2022	Rectal Cancer	A cohort of 842 LARC patients treated with standard nCRT	DeepPCR	AUC of 0.723 (95% CI: 0.591, for predicting pCR

* ER+: estrogen receptor positive, HER2+, human epidermal growth factor receptor 2 positive, *easTILs: represents the density of TILs over stroma area which mimics the pathologist scoring of sTIL per instructions from the International Immuno-Oncology Biomarker Working Group on Breast Cancer; LR: lymphocyte ratio; iTILs: intratumoral TILs; PD-L1 TPS: programmed death ligand-1 tumor proportion score; NAC: neoadjuvant chemotherapy; OSAMTL-DNS: one-step abductive multi-target learning with diverse noisy samples; LARC: locally advanced rectal cancer; AUC: area under the curve; SCD; stromal cellular density; TSR: tumor stroma ratio; ALN: axillary lymph node; cPR; complete pathological response; HR: hazard ratio.

Table 2. Overview of the possible opportunities and challenges associated with the implementation of artificial intelligence in digital pathology.

Opportunities	Challenges
Inter-pathologist subjectivity greatly reduced	Significant amount of digital histopathological specimens required for training and testing.
Capable of identifying subvisual structural characteristics	Requirement of extensive clinical data accompanying image data.
More efficient completion of laborious tasks, such as completing mitotic counts	Lack of multicentre, prospective, and randomized trials evaluating performance.
Quicker patient turnaround on completion and validation of AI model.	Computational expenses such as the requirement of GPUs.
Spatial algorithms can analyze the geographical context of data	Long term cost of data storage.
Readouts can be easily accessed remotely by pathologists	Absence of reimbursement models.
Capable of analyzing and predicting instability and mutation within histopathological images	Binary manner of classification.
Can infer information on patient immune response and treatment efficacy through analysis of immune cell location relative to tumor cells.	Lack of guidelines surrounding the annotation, image and laboratory requirements to mimic natural variation.