Artificial intelligence in digital histopathology for predicting patient prognosis and treatment efficacy in breast cancer

ABSTRACT

Introduction

Histological images contain phenotypic information predictive of patient outcomes. Due to the heavy workload of pathologists, the time-consuming nature of quantitatively assessing histological features, and human eye limitations to recognize spatial patterns, manually extracting prognostic information in routine pathological workflows remains challenging. Digital pathology has facilitated the mining and quantification of these features utilizing whole-slide image (WSI) scanners and artificial intelligence (AI) algorithms. AI algorithms to identify image-based biomarkers from the tumor microenvironment (TME) have the potential to revolutionize the field of oncology, reducing delays between diagnosis and prognosis determination, allowing for rapid stratification of patients and prescription of optimal treatment regimes, thereby improving patient outcomes.

Areas Covered

In this review, the authors discuss how AI algorithms and digital pathology can predict breast cancer patient prognosis and treatment outcomes using image-based biomarkers, along with the challenges of adopting this technology in clinical settings.

Expert Opinion

The integration of AI and digital pathology presents significant potential for analyzing the TME and its diagnostic, prognostic, and predictive value in breast cancer patients. Widespread clinical adoption of AI faces ethical, regulatory, and technical challenges, although prospective trials may offer reassurance and promote uptake, ultimately improving patient outcomes by reducing diagnosis-to-prognosis delivery delays.

KEYWORDS:

• Algorithm
• artificial intelligence
• breast cancer
• deep learning
• digital pathology
• histopathology
• image-biomarker
• machine learning

1. Introduction

1.1. Background

Tumor histology uncovers phenotypic information that is indicative of not only disease progression, but also often patient outcome [1]. Developments in artificial intelligence (AI) technology have allowed the mining of previously unexploited features from routine histology images of cancer, providing potential diagnostic, prognostic, predictive, and other clinically relevant information [1]. Human assessments of histology are subjective and are prone to difficulties in terms of reproducibility due to inter-observer variability [2].

The inherent uncertainty in traditional pathology analysis may result in inappropriate treatment and secondary complications, adversely affecting patient quality of life [3]. Therefore, computational analysis of histology imaging has received significant attention since the timely and accurate assessment of tumor histomorphology and identifying pertinent prognostic/predictive markers are crucial to personalized cancer care. The utility of AI in detecting image-based prognostic/predictive biomarkers has been demonstrated to be particularly promising for breast cancer [4–6].

Breast cancer is the most common female cancer, accounting for over a quarter of all cancers in women, with more than 2.3 million women diagnosed worldwide in 2020 [7]. Breast cancer is a heterogeneous disease that still presents challenges for clinicians in predicting the likelihood of disease progression, particularly in patients where the disease is detected in the early stages [8]. To manage the increasing volume of breast cancer cases, meaningful interpretation of breast cancer progression to establish prognostic factors and limit the number of patients undergoing unnecessary treatments is paramount.

Approximately 30% of patients initially presenting with early-stage disease develop a recurrence of the disease within 10 years and, therefore, require necessary, sometimes aggressive chemotherapy [9]. However, differentiating between early-stage breast cancer patients whose disease will or will not recur remains a clinical challenge to avoid unnecessary application of aggressive treatment strategies. Traditionally, most early-stage patients with breast cancer were treated with chemotherapy to prevent recurrence, despite many not benefiting from such treatment, thereby exposing these individuals to severe chemotherapy-related side effects and often additional financial burden [10]. Current approaches to treatment stratification and prognostic prediction of survival for breast cancer patients are based on molecular tests that are not readily available in many jurisdictions due to cost and other barriers. Thus, identifying an alternative, cost-effective patient stratification approach using AI-based digital pathology analysis of histopathology images has great potential for global patient benefit. This review aims to summarize the status of published work on prognostic/predictive image biomarkers for breast cancer that have been developed using AI and digital pathology.

1.2. What is AI, machine learning, and deep learning?

AI is a broadly encompassing term referring to the branch of computer science that attempts to emulate human intelligence. These specialized systems are developed and programmed to perform complex cognitive functions such as learning, problem-solving, and decision-making, tasks originally reserved for humans [11]. Machine learning (ML), a subfield of AI, is defined as a computational system based on a set of algorithms that attempts to examine vast amounts of data, by using multiple layers of analysis, to ‘learn’ from the data to make predictions on future data (Figure 1) [12,13]. Over time, the computational algorithm performance improves with experience in respect to the instructed task, such as classification, regression, and clustering [14,15]. Deep learning (DL) is a ML technique comprising multiple layers of artificial neural networks (ANNs), which were devised in the 1980s as artificial representations of the human neural architecture [13]. ANNs incorporate three functional layers of artificial neurons or ‘nodes,’ which include an input layer, multiple hidden layers, and an output layer (Figure 1) [11]. Each node is connected to the immediately earlier and deeper layers via edges, with the strength of their connection being referred to as their ‘weight’ [16]. The input layer nodes typically consist of the desired variables being quantified in the dataset of interest – for example, each node might represent the intensity value of a specific pixel in an image. The hidden layers of the ANN perform feature construction for the layers prior, with deeper layers refining earlier layers [16]. For this reason, these algorithms can automatically learn feature representations without feature engineering, avoiding bias and achieving end-to-end result output [17]. Improvements in computational processing power have led to an increased interest in using AI for digital pathology over the past decade.

Figure 1. The differences between traditional machine learning (ML) and deep learning (DL). AI: artificial intelligence; CNN: convolutional neural network. An input layer, an output layer, and multiple hidden layers make up convolutional networks.

1.3. Digital pathology

Digital pathology is essentially the process of generating high-resolution digital images from tissue sections on glass slides that are typically examined under a microscope, using complex, novel imaging systems and whole slide image (WSI) scanners [18–20]. WSI scanners capture multiple images of entire tissue sections within minutes, allowing the complex spatial relationships within the tumor microenvironment (TME) to be assessed digitally. Additionally, digital images can be shared between hospitals and research labs for real-time analysis (telepathology), significantly improving workflow efficiency [3]. In order to implement digital pathology successfully in the clinic, several critical elements need to be acquired namely (1) hardware, which is composed of a digital slide scanner, (2) software, which facilitates image creation, visualization, management, and analysis, and (3) network connectivity facilitated by central storage/cloud-based systems (Figure 2) [21,22]. Furthermore, ML and DL algorithms have been successfully incorporated into digital pathology workflows to computationally evaluate images, limiting the subjective nature of visual perception, human errors in data integration, and opinion between interobservers (Figure 2) [15]. Moreover, the application of AI in digital pathology allows for the integration of clinical parameters, which presents an opportunity to develop image-based biomarkers for predicting breast cancer patient prognosis and treatment efficacy, a focal point of this review paper.

Figure 2. Artificial intelligence (AI) and Digital Pathology Workflow. DIA: data-independent acquisition. Initially, tissue samples are stained to highlight key histological features, followed by the digitization of the samples into high-resolution digital images using scanners. AI-based algorithms are subsequently applied to identify and quantify relevant features (ex-TIL count, stromal, and tumor area) within the images. The final step involves evaluating the association between the identified features and patient survival outcomes through statistical analysis.

2. Application of AI in digital histopathology in breast cancer (predictive and prognostic biomarkers)

Both ML and DL algorithms have been demonstrated to successfully stratify breast cancer patients for prediction of prognosis and response to treatment. This review discusses their application in investigating the prognostic and predictive potential of histopathological features (Section 2.1), along with how DL algorithms can bypass conventional feature selection for prognosis prediction (Section 2.2).

2.1. Histopathological annotations

2.1.1. Tumor infiltrating lymphocytes

2.1.1.1. Prognostic

A successful response to immunotherapies is dependent on the immune contexture of the TME, with the presence of certain cell types, such as tumor infiltrating lymphocytes (TILs), being associated with enhanced survival in patients with various forms of cancer [23,24]. Traditionally, TILs are scored semi-quantitatively on standard H&E-stained sections as absent, non-brisk, or brisk, and some authors have suggested utilizing a four-tier grading system that considers TIL distribution and density [25]. However, despite these well-established scoring methods and the body of evidence supporting a prognostic role for this biomarker, TIL quantification is yet to be broadly adopted for clinical decision-making due to a lack of standardization between institutes and apprehension regarding reproducibility between pathologists [25]. However, the International Immuno-oncology Biomarkers Working Group has aimed to standardize TIL scoring through a guideline designed for the visual evaluation of TILs in Hematoxylin and Eosin (H&E) sections, with its application in breast cancer leading to an acceptable interobserver agreement between multiple pathologists [26–28]. Moreover, AI and digital pathology can further aid in this standardization by analyzing complex spatial patterns and providing metrics for vigorous validation to allow for reliable patient stratification [25,29–31]. As such, the prediction of patient prognosis using AI and digital pathology was demonstrated in a cohort of triple-negative breast cancer (TNBC) patients using the open-source software QuPath [32]. Interestingly, multivariate Cox regression analysis adjusting for stage, age, and histological grade demonstrated that measurement of the density of TILs over stromal area: easTIL% (100 * [sum of TIL Area (mm2)/Stroma Area (mm2)]), a marker which mimics the International TIL Working Group variable used by pathologist, was prognostically significant for overall survival (OS) in the validation cohort (p = 0.0003). Moreover, Heindl et al., demonstrated similar results in estrogen receptor positive (ER+) breast cancer using an AI image analysis pipeline exploiting the morphological differences between cancer cells, lymphocytes, and stromal cells in H&E-stained tissue sections obtained from 1,178 patients enrolled in the Arimidex or Tamoxifen Alone or Combined Trial (ATAC), 141 of which were also human epidermal growth factor receptor 2 positive (HER+) [33]. Accuracy rates for identifying the three cell types were 93.8% for cancer cells, 87.9% for lymphocytes, and 84.2% for stromal cells. The immune abundance metrics, such as overall lymphocyte infiltration and intratumor lymphocyte ratio, did not provide significant prognostic information for recurrence [33]. In contrast, high immune spatial scores, indicating increased cancer and lymphocyte spatial clustering, were significantly associated with poor 10-year recurrence-free survival. Moreover, the prognostic value for spatial scores for late recurrence (5–10 years) was similar to that of IHC 4 and Recurrence Score (RS; Oncotype Dx) in both the overall population and HER2+ population. Additionally, Immune-Cancer Hotspot, Immune Hotspot, and Cancer Hotspot scores provided statistically significant prognostic value when added to IHC4 and RS for 0–10 years, 5–10 years, and in both time windows, respectively, for the overall patient population. However, none of the combined scores outperformed the Clinical Treatment Score, a risk prediction model calculated from clinicopathological variables, and the PAM50 Risk of Recurrence Score [33]. Furthermore, in accordance with these findings, Thagaard et al., 2021 demonstrated that stromal tumor-infiltrating lymphocyte (sTILs) could be utilized as a prognostic image biomarker for TNBC using a fully automated digital image analysis pipeline [34]. The multistep process, developed and performed on the Visiopharm platform, involved 1) training a convolutional neural network (CNN) to differentiate between tissue and slide background, 2) a second level CNN that segmented areas of tumor, necrosis, and noninvasive epithelium (normal, pre-invasive lesions), 3) an object-based density analyzer which outlined tumor and tumor-associated stroma regions, 4) a third level CNN which detected TILs (as mononuclear immune cells), and 5) the generation of tumor heatmaps [34]. The algorithm was tested on 257 patients, with univariate analysis demonstrating that high sTIL density was associated with significantly prolonged OS (every 10% or 300 sTILs/mm² increase in the biomarker score resulted in a ~ 20% decrease in risk of death). Moreover, Thagaard et al., 2021 demonstrated that sTIL density was prognostic for OS, independent of age, tumor size, nodal status, and tumor type (HR: 0.81, p = 0.001) [34]. Furthermore, Makhlouf et al., 2023 utilized a supervised DL model to analyze TILs from H&E-stained WSI from a cohort of 2,231 luminal early-stage breast cancer patients [35]. The study elucidated that high sTILs and intatumoral (t)TILs counts were associated with significantly shorter survival than tumors with low sTILs and tTIL counts; HR = 2.5, 95% CI = 1.3–4.5, p = 0.004 and HR = 2.0, 95% CI = 1.06–3.7, p = 0.03, respectively [35]. Additionally, the proximity of TILs to stromal and tumor cells (co-occurrence) were associated with poor clinical outcomes and clinicopathological parameters including younger age at diagnosis, high tumor grade, large tumor size, and lymph node metastasis, signifying the effective use of AI and digital pathology to elucidate the prognostic utility of TILs in luminal breast cancer [35]. In summary, these studies illustrate the potential for AI algorithms to predict OS for breast cancer patients using TILs as an image-based biomarker in the clinic.

2.1.1.2. Predictive

Breast cancer patients are often administered neoadjuvant therapy, that is, pre-operative systemic treatment in the form of chemotherapy, radiotherapy, and endocrine therapy [36]. Patients with tumors that demonstrate a pathological complete response (pCR), a surrogate endpoint for event-free survival (EFS), have a reduced risk of recurrence compared to those achieving a partial response [37,38]. However, only 20–30% of patients achieve a pCR, with this being dependent on the tumor biology; accordingly, identifying image biomarkers using AI and digital pathology may aid in the stratification of these patients [39,40]. Lee et al., 2022 demonstrated that AI-powered spatial TIL algorithms could successfully evaluate TIL densities in the tumor area and surrounding stroma in TNBC patients [41]. Mean intratumoral (i) TIL scores (5.8% vs 4.5%, p < 0.001) and surrounding stromal (s)TIL scores (12.1% vs 9.4%, p < 0.001) were significantly higher in patients who achieved a pCR after neoadjuvant chemotherapy, with iTIL scores remaining an independent predictor of pCR along with CT stage and Ki-67 after multivariate analysis [41].

Recently, Fanucci et al., 2023 assessed the predictive value of TILs for pCR and event-free survival in patients with stage IIB-IIIC HER2$-$ breast cancer randomized to neoadjuvant chemotherapy (NAC) ± bevacizumab [42]. In these 113 pretreatment patient samples, the authors demonstrated that a ML-derived digital measure of TILs (easTILs, described above) was predictive of pCR in breast cancer where the mean pretreatment easTILs% was significantly higher in cases with pCR compared to residual disease (median 36.1 vs 14.8%, p < 0.001). Quantification of easTILs had more effective outcome discrimination compared to the sTILs score interpreted by a pathologist [42].

Further, Choi et al., 2023 established a DL-based algorithm to evaluate sTIL expression in breast cancer WSIs [43]. Following manual assessment by a pathologist, WSIs from TNBC and HER2+ patients were reevaluated by the pathologist with the assistance of the DL model. Firstly, DL-assistance increased the concordance correlation co-efficient among pairs of pathologists. Further, compared to review by pathologists alone, DL-assistance further increased the difference between the average sTIL scores allocated to responders vs non-responders to NAC (26.8 ± 19.6 vs. 19.0 ± 16.4; p = 0.003), further demonstrating the ability of AI to heighten the predictive power of TIL scoring [43].

In line with these findings, TILs have been demonstrated to predict breast cancer patient response to immune-checkpoint inhibition in a number of clinical trials, including Keynote-086 [44]. Various studies have demonstrated the utility of AI to investigate TIL-based prediction of immunotherapy response. Park et al., 2022 categorized the TME of advanced non-small cell lung cancer (NSCLC) patients by immune phenotype, as either inflamed, immune-excluded, or immune-desert, based on TIL density using an AI-powered algorithm on H&E-stained WSIs [45]. Patients with tumors exhibiting an inflamed immune phenotype demonstrated higher response rates to immune-checkpoint inhibitors, an increase in cytolytic activity and had prolonged progression-free survival (4.1 months), compared to patients whose tumors were differentiated as having an immune-excluded (2.2 months) or an immune-desert phenotype (2.4 months) [45]. Additionally, there was a significant correlation between the AI model (Lunit SCOPE IO) assessment of TILs and pathologists’ manual evaluation of TILs (p < 0.001), indicating the potential for AI-based models to predict patient response to treatment in a reliable, accurate, and time-effective manner [45]. These results demonstrate the potential use of TIL scores in predicting treatment response in breast and other cancer patients.

In 2023, Li and colleagues developed a novel 25 ML algorithm-based immune infiltrating cell (MLIIC) signature to assess TNBC prognosis, mutation, biological function, drug responsiveness, immune infiltration, and immunotherapy responsiveness [46]. In a similar study, Tang et al., 2023 utilized ML algorithms to develop an immunophenotype-based risk assessment model for TNBC patients. The model demonstrated a significant improvement in its ability to predict prognosis and drug sensitivity for TNBC patients, with a high level of accuracy [47].

2.2. Mitoses/large-scale DNA organization

2.2.1. Prognostic

In recent years, several genomic tests that predict cancer patient prognosis have been incorporated into the clinic [48]. The effects of genetic and epigenetic changes are reflected in the nuclei of cancer cells via their morphological and chromatin organization [49]. Advances in image processing algorithms, data science, optics, and microscopy have enabled the quantification of these nuclear features during tissue assessments of sectioned material by pathologists [49]. In particular, the quantification of Large-Scale DNA Organization (LDO), referring to the measurement of different characteristics of the nuclear DNA organization such as the size, shape of the nucleus, and the intensity of the staining, can be correlated with disease states to predict cancer patient prognosis [49–51]. Guillaud et al., 2018 demonstrated that LDO was an independent prognostic marker for breast cancer survival [49]. A forward stepwise linear discriminant analysis determined 6 nuclear features that classified patients into ‘survival’ and “deceased’’ groups. A total of 76.3% of cells (of 5493 total detected cells) were correctly classified (into ‘survival’ and ‘deceased’ groups) with a sensitivity and specificity of 81.2% and 73.5%, respectively. Moreover, when a core LDO value of 60% was used as a cutoff point to differentiate between LDO negative (LDO score lower or equal to 60%) or LDO positive (LDO score higher than 60%) cores, the sensitivity and specificity of this prognostic test increased to 84.4 and 81.6%, respectively [49]. In line with these findings, LDO scores were demonstrated to be predictive of prognosis in other solid tumor cancer types such as prostate [52] and lung [53], indicating the potential efficacy of this marker to be utilized in the clinic in the future.

The Nottingham Grading System has been widely used to determine the grade of breast cancer, a significant indicator of patient prognosis [54]. The system focuses on three nuclear biomarkers, including nuclear atypia, tubule formation, and mitotic cell count [55]. Among these biomarkers, mitotic cell count is considered the most important in terms of grading due to its direct relationship with tumor prognosis. However, manually counting mitotic features is time-consuming (typically taking a pathologist 5–10 min to perform), subjective, and suffers from low reproducibility [55,56]. Additionally, it may be difficult to differentiate between a mitotic figure and a cell undergoing degeneration, apoptosis, or necrosis [57]. AI coupled with digital pathology may offer a potential solution to the aforementioned problem. Although studies investigating the prognostic utility of AI-based histological analysis of mitotic figures are sparse, Pantanowitz et al., 2020 demonstrated that pathology end-users were more accurate and efficient (overall timesaving of 27.8%) at quantifying mitotic figures in digital images of invasive breast carcinoma with the aid of AI [57]. In 2023, van Bergeijk and colleagues conducted a study on AI-assisted approaches to mitotic counting (MC) in breast cancer biopsies/resections using WSI and light-microscopy (LM) [58]. The study found that WSI-MC, with the help of AI aligns well with the LM-MC. As a result, WSI-MC is considered a promising alternative to LM-MC [58]. More recently, Ibrahim et al., 2024, investigated the optimal method of using AI-based mitotic figure scoring in breast cancer [59]. This study demonstrated that mitotic activity index (MAI), defined as the number of mitotic figures in 3 mm² area within the mitotic hotspot, positively correlated with the gold standard visual scoring method used in the Nottingham Grading System (R = 0.8) and Ki67 scores (R = 0.69).

Regarding the ability of such an AI model to infer patient prognosis utilizing mitotic count, Ibrahim et al., 2024 also demonstrated that mitotic activity index (MAI) was an independent predictor of survival (p < 0.05) in multivariate Cox regression analysis [59]. Moreover, Jaroensri and colleagues utilized DL approaches to develop an AI-based Nottingham grading system [60]. The system was used to grade the three Nottingham grading system components; nuclear pleomorphism, tubule formation and mitotic count, in invasive breast carcinoma WSIs, with comparison to pathologist standards [60]. Regarding the prognostic ability of mitotic count, utilizing progression-free interval as the endpoint, the AI-determined MC achieved a C-index of 0.58 (95% CI: 0.53, 0.64) vs that obtained by the pathologist-given MC of 0.54 (95% CI: 0.48, 0.59) [60]. Further providing evidence of its prognostic potential, the AI-based model provided a MC score that demonstrated better correlation with Ki-67 gene expression with a correlation co-efficient of 0.47 (95% CI: 0.41, 0.52) vs a correlation coefficient of 0.37 (95% CI: 0.32, 0.43) between pathologist-based MC and Ki-67 gene expression, which may suggest that the AI-based findings could potentially be more indicative of the biologically relevant ground truth i.e. cell proliferation [60]. These studies represent a potential future opportunity to utilize AI algorithms for predicting treatment outcomes in breast cancer patients based on mitotic analysis.

2.3. Nuclear features

2.3.1. Prognostic

As the field of digital pathology advances, nuclear architecture has arisen as a potential prognostic and predictive morphological feature in breast cancer. Not only could variations in nuclear shape reflect genetic instability but they may also influence the ability of cancer cells to migrate through tissues and form metastatic deposits that contribute to recurrence [61,62]. Beck et al., 2011 developed a ML-based model (C-Path) for automatically analyzing nuclear and cytoplasmic features from H&E-stained images to predict 5-year survival in two independent cohorts of breast cancer patients (Netherlands Cancer Institute (NKI): n = 248; Vancouver General Hospital (VGH): n = 328) [63]. It was demonstrated that C-Path scores were significantly associated with OS (log-rank p = 0.001) in the test VGH dataset, in comparison to the standard manual pathological grading score assigned to WSIs from the same cohort of patients (log-rank p = 0.29) [63]. Furthermore, Bootstrap analysis on the NKI dataset revealed that the measure of the relative border between spindled stromal nuclei to round stromal nuclei robustly contributed to the C-Path model, with an increased relative border of spindled stromal nuclei to round stromal nuclei being associated with worse prognostic outcomes [63]. Although the biological significance of this morphological feature remains to be elucidated, this observation highlights a potential prognostic relationship between different stromal cell populations in breast cancer.

2.3.2. Predictive

Previous studies have also demonstrated that nuclear features, such as the modified Bloom Richardson (mBR) scale [64], nuclear grade [65], mitotic count [66], and tubule nuclei density [67], highly correlated with Oncotype DX risk category. Whitney et al., 2018 used supervised ML classifiers to catalog digitalized H&E ER+ breast cancer images into Oncotype DX risk categories through nuclear morphological feature evaluation [62]. Interestingly, histomorphometrically, low Oncotype DX and intermediate Oncotype DX cases appeared more similar compared to the high Oncotype DX cases [62]. Moreover, nuclear architecture and the shape of stromal nuclei were identified as the most discriminating epithelial and stromal features, respectively, when stratifying the different risk categories. These morphological features were also assessed by Lu et al., 2018, who investigated whether a computer-extracted measurement of nuclear orientation combined with features relating to nuclear shape and texture were predictive of aggressiveness and prognosis in H&E-stained images from a cohort of 276 early-stage ER+, lymph node negative (LN$-$) breast cancer patients [68]. The image classifier was constructed on an initial modeling group (n = 177) using 12 nuclear orientation features, a measure of the disorder in the orientation of adjacent nuclei, with increased heterogenicity of nuclear orientation being associated with worse survival outcomes reflecting the poorly differentiated, disorganized cell growth observed in aggressive tumors (AUC = 0.67 ± 0.03) [68]. In an independent validation cohort (n = 99), the predictive labels generated by the image classifier correlated with worse survival independent of other variables (HR 3.17, 95% CI: 0.33–30.46, p = 0.01039) [68]. These results highlight a potential prognostic impact of nuclear features when stratifying ER+, LN$-$ breast cancer patients into low and high recurrence risk categories.

The evaluation of tumor nuclear features using AI has also been demonstrated to predict response to treatment in breast cancer. Doddington et al., 2021 developed multiple deep CNNs to automate tumor detection and nuclear segmentation in a cohort of 58 HER2+ and TNBC patients (pCR; n = 37, pathological partial response (pPR); n = 21) to predict response to NAC [69]. Features such as nuclear count, area, circularity, and image-based first- and second-order features including mean pixel intensity and correlation of the Gray-Level Co-Occurrence Matrix (GLCM-COR) were assessed. Multivariate analysis showed that higher nuclear intensity (OR = 1.23, p = 0.018), fewer multifocal/multicentric tumors (OR = 0.14, p = 0.012), and lower GLCM-COR (OR = 0.96, p = 0.043) were each independently associated with the likelihood of achieving a pCR, with a model combining all three features successfully predicting treatment response in 79% of cases (62% for pPR and 89% for pCR) [69]. Saednia et al., 2022 demonstrated similar findings in respect to the predictive efficacy of tumor nuclear features, with their best performing ML algorithm, incorporating GLCM and graph-based features extracted from tumor nuclear masks, achieving a test accuracy of 84%, a sensitivity of 85%, a specificity of 82% and an AUC of 0.9 [70]. More recently, Shen et al., 2023, developed a novel AI pipeline to investigate cancer atypia that utilizes three independent models, including support vector machine (SVM) and random forest models that learn nuclear atypia from fine-grained nuclear features extracted by image analysis method [71]. The CNN model successfully predicted response to NAC in a cohort of 103 breast cancer patients with an accuracy of 95.2% [71]. These studies indicate the potentially predictive significance of tumor nuclear features when combined with digital pathology and AI algorithms in breast cancer.

2.4. Tumor-Stroma ratio

2.4.1. Prognostic

The TME is well known to play a pivotal role in the evolution of malignancies, particularly in the context of solid tumors. The complex organization of numerous cell types (such as adipose, immune, neuroendocrine, and fibroblast cells), blood and lymph vessels, and the extracellular matrix, makes for an environment that can widely influence the hallmarks of cancer [72]. In particular, the tumor-stroma ratio (TSR) has come to light as a prognostic biomarker for solid tumors, with high stromal content (comprising cancer-associated fibroblasts) being associated with poorer outcomes [73]. In fact, TSR was demonstrated to have prognostic impact in 16 solid tumor types (2,732 cancer patients) as shown by Micke et al., 2021, who calculated the ratio using immunofluorescent staining for epithelial markers and ML image analysis algorithms [74]. In the context of breast cancer, Yan et al., 2022 developed a modified TSR assessment algorithm using the open-sourced image-analysis platform QuPath to assess the prognostic utility of TSR in 240 breast cancer patients [75]. Tissue microarrays (TMAs) were stained with cytokeratin immunohistochemistry (IHC) to distinguish between the stromal and tumor areas prior to the application of automated quantitative image-analysis algorithms [75]. Statistical analysis revealed that increased stromal content (low TSR) was associated with worse 5-year disease-free survival (DFS) in the complete breast cancer cohort (p < 0.0001), and when breast cancer was stratified into HER2+ (p < 0.05), TNBC (p < 0.05), and luminal HER2$-$ (p < 0.05) subtypes [75]. Interestingly, TSR was also significantly associated with ER (p = 0.005) and progesterone receptor (PR) status, (p = 0.035), and strongly associated with tumor size (p = 0.051) [75]. These results demonstrate the prognostic impact of TSR in different subtypes of breast cancer [75]. Similar findings were highlighted by Millar et al., 2020, who quantified TSR in digital images of H&E-stained TMAs in a cohort of 647 luminal and TNBC patients [76]. The study demonstrated that in TNBC, low TSR (i.e. high stroma) was associated with OS (OS; HR 1.90, CI 1.10–3.29, p = 0.021) and breast cancer-specific survival (BCSS; HR 2.64, CI 1.31–5.35, p = 0.007) following multivariate analysis after consideration of age, size, grade, sTILs, lymph nodal status and chemotherapy [76]. Additionally, when TSR was combined with the degree of TIL (Low ≤30% ≥ High) improved prognostic performance was demonstrated, with the high TSR/high TILs cohort having a 69% reduction in the risk of death compared to the low TSR/low TILs cohort for TNBC. In contrast, in luminal breast cancer patients, low TSR (i.e. high stroma) predicted favorable outcomes for OS in multivariate analysis independent of age, size, grade, sTILs, lymph nodal status, endocrine therapy and chemotherapy. In terms of BCSS, low TSR was a significant predictor of survival in univariate analysis but not multivariate analysis in luminal tumors [76]. These results not only reflect the potential clinical significance of TSR in predicting breast cancer patient prognosis, but also highlight the inherent differences between the stroma of different subtypes of breast cancer. Although the biological significance of this morphological feature is yet to be elucidated, the findings from this study highlight the prognostic significance of spatial relationships between different stromal cell types [63].

Furthermore, the spatial relationships within the TME were investigated by Abubakar et al., 2021 who applied ML algorithms to an ethnically diverse collection of digitalized H&E sections, comprising 2,084 breast cancer patients from Poland, the United States (US), and China, to determine the prognostic impact of tumor-associated stromal-cellular density (SCD) [77]. SCD, defined as the percentage of tumor-stroma that is occupied by nucleated cells, was demonstrated to be an independent predictor of poor clinical outcomes in luminal cancers patients from all three study populations following multivariate analysis, with high SCD being associated with more aggressive tumors. Furthermore, SCD demonstrated prognostic power beyond standard clinical parameters such as tumor grade, size, subtype, and lymph node status [77]. The prognostic impact of the parameter may be related to the extent of cancer-associated fibroblast (CAF), TIL, and macrophage infiltration which act as a proxy for inflammation [78].

2.4.2. Predictive

Along with demonstrating prognostic utility, increasing evidence suggests that tumor–stroma interactions significantly influence treatment resistance, and therefore patient outcomes, in several cancer types [79,80]. Although studies investigating the predictive value of the tumor–stroma relationship using ML in the context of breast cancer are sparse, Li et al., 2022 developed and applied DL algorithms to predict pCR to NAC in a multicenter dataset comprising of 1,035 patients [81]. The algorithm generated a tumor-associated stroma (TS) score which predicted pCR with an AUC of 0.745, 0.673, and 0.725 in the three validation cohorts. Furthermore, WSI with high TS-scores were associated with greater lymphocyte infiltration, while WSI with lower TS scores typically had a higher abundance of collagen and fibroblasts [82]. In line with these findings, pCR rates for collagen-dominant type stroma, fibroblast-dominant type stroma, and lymphocyte-dominant type stroma were 14.2%, 20.6%, and 42.1%, respectively. These results suggest that information contained within the tumor-associated stroma can be utilized to predict response to NAC, and that stroma with high collagen and fibroblast dominance is associated with a poor response to treatment [82,83].

2.5. Automatically extracted histopathological features using DL

2.5.1. Prognostic

Unlike ML algorithms which require manual feature extraction, DL algorithms possess the ability to automatically extract histopathological features for prognosis prediction. Xu et al., 2021 developed and validated a DL-based primary tumor biopsy signature that successfully predicted axillary lymph node (ALN) metastasis preoperatively in early-stage breast cancer patients who are ALN negative (AUC 0.816) [84]. A DL model to predict breast cancer recurrence was also developed by Phan et al., 2021, who adopted a transfer learning approach that circumvented the need for region of interest labeling (AUC 0.87) [85]. Yang et al., 2022 created a deep CNN that evaluated the risk of relapse and metastasis in HER2+ breast cancer patients using H&E images [86]. The model achieved an AUC of 0.72 when independently tested on 123 HER2+ breast cancer patients from The Cancer Genome Atlas, a cohort with ethnic diversity [86]. Lazard et al., 2022 utilized DL algorithms to predict homologous recombination (HR) status, a marker of platinum salt and polyADP-ribose polymerase inhibitor chemotherapies, in luminal breast cancers from WSIs [87]. More recently, Wahab et al., 2023, developed a DL prognostic algorithm for early-stage luminal HER2$-$ BReAst CanEr, termed BRACE marker [88]. The BRACE marker successfully stratified luminal breast cancer patients for both distant metastasis-free survival (p = 0.001, C-index: 0.73) and breast cancer-specific survival (p < 0.0001, C-index: 0.84), demonstrating prediction accuracy comparable to Nottingham Prognostic Index and Magee scores, which are both derived from manual histopathological assessment [88]. Additionally, DL models have been demonstrated to predict breast cancer recurrence and metastasis by integrating histological H&E images with clinical information and gene expression data (AUC 0.75), highlighting the potential for AI algorithms to not only predict prognosis through histological images but incorporate prognostic information for other modalities which may have additive prognostic power [89].

2.5.2. Predictive

DL algorithms can also be utilized to predict patient response to treatment. Li et al., 2021 developed a DL model that successfully predicted pCR to NAC from H&E images in a cohort of 540 breast cancer patients, with the model achieving an AUC of 0.822, an accuracy of 0.853, and an F1 score of 0.503 [82]. Moreover, when sTILs and breast cancer subtype were incorporated into the model, an AUC of 0.890 was attained, surpassing the performance of the sTILs-subtype model by 0.051 (0.839, p = 0.001) [82]. In line with these findings, Farahmand et al., 2022 developed a DL-CNN that predicted Trastuzumab response in HER2+ breast cancer patients with an AUC of 0.8 [90]. Furthermore, a recent study by Hoang et al., 2022 demonstrated that Deep Pathology for Treatment (DeepPT), a DL model that predicts gene expression profiles directly from histopathological images, could be used in combination with ELIGHT, a platform that predicts patient response to treatment based on their tumor transcriptomics, to predict therapy response in breast and lung cancer patients [91]. Moreover, Huang et al., 2023 utilized the DeepLabV3 model to predict NAC responses in a cohort of 62 HER2+ and 64 TNBC patients from H&E and multiplex IHC (PD-L1, CD8+, and CD163+) images [92]. These results demonstrate, not only that DL algorithms can be applied to H&E images in a prognostic and predictive manner, but that genomic and proteomic data can be inferred from these images to provide a deeper understanding of the relationships within the TME.

3. Challenges of using AI in digital histopathology for cancer research

3.1. Algorithm

Although AI holds the potential to revolutionize the digital pathology field by overcoming subjective visual assessments, reducing the burdensome and time-consuming workflow of Pathologists, thereby improving time to diagnosis/prognosis delivery, there are several challenges that must be considered prior to its implementation (Tables 1 and 2).

Table 1. Digital pathology related artificial intelligence models in oncology.

Biomarkers	Purpose	Author	Cancer Type	Dataset	Model	Performance
Tumour Infiltrating Lymphocytes (TILs)	Prognostic	Bai et al., 2021	TNBC	Discovery set: 171 patients Validation sets: 749 patients in total	Neural Network (CNN11)	High easTILs (≥19.9): HR:0.57, CI: 0.42–0.78, p < 0.0003.
		Heindl et al., 2018	ER+ Breast Cancer	Cohort of 1,178 postmenopausal patients treated with 5 years tamoxifen or anastrozole.	Fully automated H&E-stain image analysis algorithm	Immune Hotspot LR-χ2 = 14.06, p < 0.001_for 0–10 years; LR-χ2 = 6.24, p = 0.01 for 0–5 years; LR-χ2 = 7.89, p = 0.005 for 5–10 years.
		Thagaard et al., 2021	TNBC	Cohort of 257 patients	Multiple CNNs trained with a VGG-based encoder	Overall survival (HR: 0.81 CI: 0.72–0.92 p = 0.001)
	Predictive	Lee et al., 2022	TNBC	Cohort of 954 patients	Lunit SCOPE IO	iTIL score was found to’ultivariable analysis (adjusted odds ratio 1.211 (95% CI 1.125–1.304) per 1 point (%) change in the score, p < 0.001).
		Park et al., 2022	NSCLC	Development: 3,166 WSI of 25 cancer types Validation: lung adenocarcinoma (LUAD, n = 461) and lung squamous cell carcinoma (LUSC, n = 462) images from The Cancer Genome Atlas (TCGA), as well as NSCLC primary tumor tissues from Samsung Medical Center (SMC, n = 1,205) and Seoul National University Bundang Hospital (SNUBH, n = 261)	Lunit SCOPE IO	In the PD-L1 TPS 1%-49% subgroup, the AUROC for prediction of tumor response to ICI by IP was 0.7609, and the AUROC by PD-L1 TPS status was 0.5561 (p < .05).
Mitosis/LDO	Prognostic	Guillaud et al., 2017	Breast	A cohort of 95 patients	Forward stepwise linear discriminant analysis	Logrank test: p < 0.0001
	Predictive	Zhao et al., 2021	Colorectal	A cohort of 496 patients with stage I, II, or, III	GLEM4D	Stage II patients with at least one high-risk factor (non-diploidy, high-stroma, chromatin heterogenous), patients who received adjuvant therapy showed significantly longer OS (72.1% vs 48.3%, p = 0.007) and DFS (64.5% vs 43.9%, p = 0.015)
Nuclear Features	Prognostic	Beck et al., 2011	Breast	Two independent cohorts comprising a total of 576 patients	C-Path	Overall Survival (log-rank: p = 0.001)
	Predictive	Whitney et al., 2018	Breast	A cohort of 178 early-stage ER+ breast cancer	Random Forest, Neural Network, Support Vector Machine, and Linear Discriminant Analysis, via 3-fold cross validation.	Low Oncotype DX (ODx) risk categories vs. High ODx risk categories (AUC = 0.72)
		Lu et al., 2018	Breast	A cohort of 278 early-stage ER+, LN breast cancer patient	Linear discriminant analysis classifier	Multivariate analysis: hazard ratio (95% confident interval) = 3.17(0.33–30.46), p = 0.01039.
		Doddington et al., 2021	HER2+ and TNBC	Fifty-eight HER2+ or TNBC patients	Multiple Deep CNN	Model successfully classified 79% of cases (62% for pPR and 89% for pCR)
		Saednia et al., 2022	Breast	A cohort of 149 patients treated with NAC	weighted U-Net model.	NAC response prediction at pre-treatment: AUC of 0.90, sensitivity of 85%, and a specificity of 82%
Tumor/Stroma Ratio	Prognostic	Yan et al., 2022	Breast	A cohort of 240 patients	Open CV	Low vs. High TSR for predicting DFS: p < 0.0001
		Millar et al., 2020	Luminal and TNBC	A cohort of 403 luminal and 244 TNBC	QuPath	Low TSR for TNBC: OS (HR 1.9, CI 1.1–3.3, p = 0.021); Low TSR for Luminal breast cancer: (HR 0.6, CI 0.4–0.8, p = 0.001)
		Abubakar et al., 2021	Breast	An ethnically diverse group of 2,084 breast cancer patients from China, Poland, and the United States	HALO	SCD was predictive of poor clinical outcomes: multivariable HR (95%CI) fourth(Q4) vs first(Q1) quartile = 1.86(1.06–3.26);Ptrend = 0.03), Poland (HR(95%CI)Q4 vs Q1 = 1.80(1.12–2.89);Ptrend = 0.01), and United States (HR(95%CI)Q4 vs Q1 = 2.42(1.33–4.42);Ptrend = 0.002).
	Predictive	Li et al., 2022	Breast	A multicentre dataset of 1035 patients	OSAMTL-DNS	TSR predicted pCR with an AUC of 0.745, 0.673, and 0725 in the validation cohorts
Deep Learning	Prognostic	Xu et al., 2021	Breast	A cohort of 1,058 breast cancer patients	attention-based multiple instance-learning (AMIL) framework	Predicting ALN metastasis: AUC of 0.816 (95% confidence interval (CI): 0.758, 0.865)
		Phan et al., 2021	Breast	A cohort of 138 patients	VGG16, ResNet50, ResNet101, Inception_ResNet, EfficientB5, and Xception	Accuracy of models for predicting recurrence: 0.84, 0.85, 0.83, 0.84, 0.87, and 0.91, respectively.
		Yang et al., 2022	HER2+ Breast Cancer	A cohort of 127 HER2+ breast cancer patients	Deep CNN	AUC of 0.76 for predicting recurrence and metastasis risk
	Predictive	Li et al., 2021	Breast	A cohort of 540 breast cancer patients receiving NAC	CNN	AUC of 0.847 in predicting pCR directly, and achieved accuracy, F1 score, and AUC of 0.853, 0.503, and 0.822
		Farahmand et al., 2022	HER2+ Breast Cancer	A cohort of 187 HER2+ breast cancer patients who received neoadjuvant targeted therapy with trastuzumab± pertuzumab prior to definitive surgery was used for training.	Inception v3 architecture	AUC of 0.80 for predicting trastuzumab response
		Lou et al., 2022	Rectal Cancer	A cohort of 842 LARC patients treated with standard nCRT	DeepPCR	AUC of 0.723 (95% CI: 0.591, for predicting pCR

* ER+: estrogen receptor positive, HER2+, human epidermal growth factor receptor 2 positive, *easTILs: represents the density of TILs over stroma area which mimics the pathologist scoring of sTIL per instructions from the International Immuno-Oncology Biomarker Working Group on Breast Cancer; LR: lymphocyte ratio; iTILs: intratumoral TILs; PD-L1 TPS: programmed death ligand-1 tumor proportion score; NAC: neoadjuvant chemotherapy; OSAMTL-DNS: one-step abductive multi-target learning with diverse noisy samples; LARC: locally advanced rectal cancer; AUC: area under the curve; SCD; stromal cellular density; TSR: tumor stroma ratio; ALN: axillary lymph node; cPR; complete pathological response; HR: hazard ratio.

Table 2. Overview of the possible opportunities and challenges associated with the implementation of artificial intelligence in digital pathology.

Opportunities	Challenges
Inter-pathologist subjectivity greatly reduced	Significant amount of digital histopathological specimens required for training and testing.
Capable of identifying subvisual structural characteristics	Requirement of extensive clinical data accompanying image data.
More efficient completion of laborious tasks, such as completing mitotic counts	Lack of multicentre, prospective, and randomized trials evaluating performance.
Quicker patient turnaround on completion and validation of AI model.	Computational expenses such as the requirement of GPUs.
Spatial algorithms can analyze the geographical context of data	Long term cost of data storage.
Readouts can be easily accessed remotely by pathologists	Absence of reimbursement models.
Capable of analyzing and predicting instability and mutation within histopathological images	Binary manner of classification.
Can infer information on patient immune response and treatment efficacy through analysis of immune cell location relative to tumor cells.	Lack of guidelines surrounding the annotation, image and laboratory requirements to mimic natural variation.

3.1.1. Lack of annotated data for algorithm training (small data sets, shortage of pathologists, image quality)

In order to train an AI algorithm, a large sample of annotated (i.e. ‘labelled’) high quality images must be available. These labels delineating the region of interest (ROI) in a tissue section should ideally be curated by expert histopathologists whom may be challenged, not only by time constraints, but also by low resolution images, ambiguity of features, and slow network connectivity [93]. Crowdsourcing is an alternative option to speed up the acquisition of labeled data, however, it possesses the ability to introduce interobserver variability. Furthermore, the curation of labeled data can also be limited by small sample size. In this case, there are several computational approaches that can be employed to maximize the utility of the available samples, such as data augmentation and active learning.

3.1.2. Algorithm validation for integration into the real world

For wide-spread integration of AI into digital pathology workflows, ML, and DL algorithms need to be extensively validated using multi-institutional data to ensure reproducibility of predictions [13]. Variations in slide preparation, scanner models, and image format and analysis all highlight the need for standardized protocols and data normalization techniques to prevent false positive and negative predictions [11,13]. Moreover, different AI models are trained with different datasets of varying cohort sizes, with larger datasets generating more accurate predictions. Therefore, standardizing data format and normalizing data analysis can not only increase generalizability but also can permit the creation of a merged universal dataset which can be used to retrain models to account for the natural variation that exists between different populations [11]. However, no official guidelines are currently available on the number of annotations, images, and laboratories required to capture this natural variation [18]. However, a protocol for the development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) has been published [94]. Standardized practices have been developed by the Digital Imaging and Communications in Medicine (DCOM) for other digital images such as X-rays, CTs, ultrasounds, MRIs, with supportive guidelines being outlined for WSIs [95].

3.1.3. Lack of transparency and interpretability (‘Black Box’)

The majority of AI algorithms in the context of digital pathology classify samples in a binary manner, that is, the model output is either a ‘yes’ or a ‘no,’ ‘malignant’ or ‘benign,’ ‘recurrence’ or ‘non-recurrence’ etc [93]. In contrast, real world prognosis delivery by a pathologist encompasses first-hand experience, cognition, clinical context, rational, and descriptive language whereby complex or rare cases are flagged and discussed at multidisciplinary team meetings [93]. These factors highlight the need for the inclusion of a plethora of heterogenous samples in the training set and incorporation of clinical data to improve model decision making. Additionally, the rationale behind sample classification is ambiguous in the context of DL, with algorithms often being referred to as ‘black boxes,’ leading to problems with transparency. This may hinder adoption of AI in the medical setting as clinicians typically justify clinical decisions to patients, which may ultimately affect the patients’ trust.

3.2. Software

3.2.1. Affordability of required computational expenses

A major obstacle to widespread adoption of AI-based digital pathology in clinical settings is the cost of the associated hardware, software, and storage [18,96,97]. The application of deep-learning in AI-based digital pathology relies on the use of specialized hardware. Graphical Processing Units (GPUs), which describe intricate electronic circuits that work to process digital images in AI algorithm training [98]. This often requires the use of specialized computers, as accomplishing AI-based activities on ordinary computers with Central Processing Units often proves slow and impractical. However, the installation and maintenance of a GPU server cluster solely to support the AI activities in a pathology laboratory is extremely costly [98,99].

The cost of long-term data storage of digital images for the development of AI algorithms also poses a significant challenge to the implementation of AI-based procedures [97,100]. WSIs account for hundreds of gigabytes of space and the training data alone must include the many variants of a diagnostic feature as well the varying depths of staining shades noted across different datasets incorporating thousands of WSIs [101,102]. While there is an availability of cloud-based storage solutions, many hospitals depend on internal data storage, and so can affect the cost of long-term storage [102]. Further compounding the financial burden is the lack of standardized reimbursement procedures [96]. While reimbursement codes are accessible for some computational analyses’ costs in countries such as the United States, they are not widely utilized and are regularly rejected [103,104]. Reimbursement models must be developed to incentivize pathology laboratories to incorporate AI into their clinical practice [104].

4. Expert opinion

4.1. Scientific and regulatory hurdles for wide-spread adoption

A lack of value determination and infrastructural support pertaining to AI and digital pathology renders individual laboratories responsible for value interpretation, investment, and cost savings considerations, which may hinder widespread adoption [18]. Furthermore, the sharing of sensitive patient data between clinics, laboratories, digital biobanks, and data scientists can enhance precision medicine while simultaneously increasing data vulnerability [11]. Therefore, it is paramount that data governance, including ethics policies, be created with guidelines on providing informed consent for patients, particularly in relation to the digital data being used to train ML and DL models [102]. Additionally, in order for the wide-spread adoption of AI and digital pathology into the clinic, pathologists must be upskilled in the area of computational pathology, particularly in relation to statistical analysis and data mining, so that if new image-based biomarkers are identified pathologists possess the skills required to fine-tune algorithms accordingly [11].

4.2. Long-term goals for AI in digital pathology

With the rapid development of technological innovations in health care, areas such as digital pathology are being increasingly utilized in numerous specialties such as oncology. The successful incorporation of AI platforms, models, and systems into routine pathology practice requires a multifaceted approach encompassing technological, regulatory, and educational considerations [105]. Robust infrastructure and computational resources are essential for processing and analyzing large volumes of pathology data efficiently. Moreover, interoperability standards and seamless integration with existing laboratory information management systems (LIMS) are crucial to ensure smooth workflow integration. From a regulatory perspective, compliance with data privacy and security regulations such as the General Data Protection Regulation (GDPR) in Europe or the Health Insurance Portability and Accountability Act (HIPAA) in the United States is paramount to safeguard patient confidentiality and data integrity. Furthermore, validation and standardization of AI algorithms is essential to ensure their reliability and reproducibility in the clinical setting, requiring collaboration between regulatory bodies, industry stakeholders, and academic institutions. In addition, clinicians, pathologists, and laboratory staff require comprehensive education and training tools to become familiar with AI tools, understand their capabilities and limitations, and effectively interpret AI outputs. As the field of AI and digital pathology develops, continuous professional development initiatives and interdisciplinary collaboration between pathologists, computer scientists, and data analysis is warranted to foster the innovative environment required for successful adoption of AI into the clinic. It is likely that clinical adoption of AI in digital pathology will begin as a tool to aid pathologists in diagnosing disease by assessing the characteristics of cancer cells, analyzing structural changes in lesions, and examining the expression pattern of biomarkers, leading to more accurate diagnoses, and faster time to prognosis prediction [106]. In line with this, an AI model (PAIGE Prostate) which detects and localizes prostate cancer in needle biopsy specimens has been approved for clinical use by the U.S Food and Drug Administration (FDA) [107]. Nonetheless, as the digital database expands and algorithm accuracy increases, AI may be utilized for more complex tasks such as predicting disease aggressiveness, patient prognosis, and treatment response [106]. ML and DL algorithms may even be used to identify the original site of cancers of unknown primary (CUP) origin [108]. Over time, genomic and proteomic data may be integrated with whole-slide imaging data to lead to the rapid expansion of AI-based computational pathology. Despite the many challenges, AI and digital pathology have extraordinary potential to significantly improve clinical workflows, increasing efficiency of clinical decision making, ultimately benefiting patients.

The adoption of digital pathology into the clinic and generation of large digital databases offers tremendous opportunity for computational analysis using AI. AI and digital pathology possess the capability to further elucidate the complex spatial relationships present in the TME and determine their diagnostic, prognostic, and predictive value. This review highlights the utility of AI to predict both breast cancer patient prognosis, and response to treatment based on several image-based biomarkers. Despite its potential, many ethical, regulatory, structural, and technical challenges remain prior to wide-spread clinical adoption of AI into pathology workflows. Prospective trials investigating the clinical benefit of AI and digital pathology may provide reassurance to both clinicians and patients, increasing trust and promoting its uptake in clinical practice, thereby reducing the delay between diagnosis and prognosis delivery and subsequentially improving patient outcomes.

A single H&E WSI generated from tumor biopsies withholds a plethora of prognostic and predictive information in the form of image-based biomarkers, some of which are discussed in the body of this review. Without the incorporation of AI algorithms to digital pathology workflows, the pragmatic approach would be to omit these biomarkers from clinical decision-making as pathologists’ workloads are too burdensome and their time too limited for manual assessment. Currently, there are several commercially available assays and standardized laboratory tests that predict breast cancer patient prognosis, such as Oncotype DX Recurrence Score, MammaPrint, Prosigna, EndoPredict, IHC4+ C-Score, and Breast Cancer Index [48]. It is evident that a considerable number of these assays are prohibitively expensive and necessitate centralized laboratory infrastructure, skilled technicians, and specialized equipment, which significantly restricts their availability in countries with limited resources. The widespread application of AI algorithms to digital H&E-stained WSIs, has the potential to ameliorate these problems, revolutionizing the pathologist’s workflow by automating tedious time-consuming tasks, improving time efficiency, and reducing workload burden.

Article highlights

• Hematoxylin and Eosin tumor images contain a plethora of histopathological information.

• The advancement of digital pathology has allowed the application of artificial intelligence algorithms to analyze the tumor microenvironment.

• Artificial intelligence algorithms can be applied to analyze histopathological images for prognostic and predictive information.

• Overcoming the challenges associated with artificial intelligence can allow for its widespread adoption in the field of oncology.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgments

We regret that we could not cite all the relevant papers in the research field due to space limitations.

Additional information

Funding

This paper was funded by: Irish Cancer Society (Collaborative Cancer Research Centre BREAST-PREDICT (CCRC13GAL; https://www.breastpredict.com), Musgrave Breakthrough Cancer Research PhD Scholarship (MUSBCR20017) North-South Research Programme administered by the Higher Education Authority on behalf of the Department of Further and Higher Education, Research, Innovation and Science and the Shared Island Fund (AICRIstart: A Foundation Stone for the All-Island Cancer Research Institute (AICRI): Building Critical Mass in Precision Cancer Medicine https://www.aicri.org/aicristart). The Science Foundation Ireland Investigator Programme (OPTi-PREDICT; 15/IA/3104); Strategic Partnership Programme Precision Oncology Ireland (18/SPP/3522; https://www.precisiononcology.ie); Frontiers for the Future Programme (19/FFP/6772) and Research Infrastructure Program (21/RI/9787; https://www.aicri.org/naspro).