Virtual tissue staining in pathology using machine learning

KEYWORDS:

• virtual histology
• virtual staining
• IHC
• pathology
• deep learning

1. Introduction

Pathology is a medical discipline dealing with diagnosing and studying diseases. Through recognizing structural histological alterations, pathologists acquire valuable information on the effect of these changes on cellular and tissue function. Pathologist evaluation is performed by examination of histologically stained tissue mounted on a glass slide through an optical microscope, or, in recent years, of a digitized version of the histological image (i.e. digital whole slide imaging, WSI).

The long-established pathology workflow consists of a series of processes to prepare stained tissue samples, which involve fixation, processing, embedding, sectioning and staining [1]. Staining is used to highlight important features of the tissue, as well as to enhance tissue contrast. This is in general a time-consuming, laborious process that needs to be performed in a designated lab infrastructure by trained technicians due to the toxicity of most chemical staining reagents. The semi-automated or manual staining processes and the utilization of different chemical reagents lead to high technical variability in sample preparation, which sometimes causes diagnostic challenges.

Furthermore, the staining process distorts the tissue and prohibits additional staining of any specific section and further molecular analysis on the same section. This is highly important in small tissue biopsies of diagnostically challenging cases, where multiple stains are often needed, followed by ancillary tests (e.g. DNA/RNA sequencing) that may be required to reach a diagnosis. If all the tissue biopsy material is used for staining, such molecular analysis cannot be performed.

2. Advancing pathology through virtual staining

Numerous optical imaging methods with alternative contrast mechanisms have been explored over the last decades [2]. Most of these were targeted at eliminating the tissue fixation step and providing an intraoperative or bedside instrument that can be used for tumor margin assessment during surgery [3]. Some of these methods have also augmented their results with a post-processing step that generates hematoxylin and eosin stain (H&E)-like images [4,5]. H&E stain is the principal stain in pathology, resulting in high contrast for the nucleus, cytoplasm, and extracellular constituents. However, the quality of these pseudo-H&E images usually lags behind the quality that pathologists are accustomed to work with, and they have limitations in representing pigments [6]. Furthermore, this type of pseudo staining is in general much more challenging to successfully apply to other types of stains beyond H&E.

Recently, our research group introduced a method to virtually stain autofluorescence images of unstained tissue sections, eliminating the need for chemical staining while creating high-quality virtually stained slides (see Figure 1) [3,7]. In this virtual staining workflow, we acquired autofluorescence images of unstained tissue slides using a conventional fluorescence scanning microscope. Next, we used a deep convolutional neural network (CNN) which was trained using the concept of Generative Adversarial Networks (GAN) [8] to learn the transformation from a label-free unstained autofluorescence input image to the corresponding bright-field image of the histologically stained version of the same sample. The network output created virtually-stained images that were well matched to the images of the same tissue samples that were labeled with different stains, including H&E, Jones, Masson’s trichrome and PAS (Periodic acid–Schiff). The virtually stained slides achieved a high degree of agreement with the histologically stained images of the same samples when they were blindly evaluated by a group of pathologists, while significantly shortening the staining time, manual labor and cost associated with lab-based chemical staining of tissue. This virtual staining framework was also applied to other contrast mechanisms, including holography and quantitative phase imaging [9] as well as reflectance confocal microscopy [10].

Figure 1. Illustration of virtual histology workflow using deep learning. The unstained tissue slide is imaged by a standard optical microscope with different fluorescence channels (DAPI/FITC/TxRed/CY5, grayscale images). A GAN-based CNN transforms the unstained tissue slides into a virtually stained image [a-Jones stain], a digital blend of histological stains [b-H&E, Jones and Masson’s trichrome stains], or IHC stain [c-HER2 stain].

The computational nature of this deep learning-based virtual staining technology enabled us to generate stains that would be impossible to create using traditional histochemical staining [11]. For example, we used what we refer to as a ‘digital staining matrix’ which allowed us to generate and digitally blend multiple stains using a single deep neural network by specifying which stain should be performed at the pixel level [11]. Not only could this novel framework be used to perform multiple stains on a single tissue section, it could also be used to create micro-structured stains, digitally staining different areas of label-free tissue with different stains. Furthermore, this digital staining matrix enabled these stains to blend together by setting the encoding matrix as a mixture of the possible stains. This technology can be used to focus on the most relevant information possible from the various virtual stains being performed so that pathologists can get more diagnostically relevant information while reducing their slide examination time.

Stemming from the success of virtual staining of the label-free tissue sections, we also created neural networks that enabled stain-to-stain transformations [12,13]. This deep learning-based framework enables the digital transformation of existing images of tissue biopsy stained with one type of stain into many other types of stains. A stain-to-stain transformation process takes less than one minute per tissue sample, as opposed to several hours or even more than a day when performed by human experts, without requiring additional tissue sections. This speed advantage enables faster preliminary diagnoses that require special stains, while also providing significant cost savings.

Another recent advancement through our virtual staining methodology was the introduction of virtual immunohistochemistry (IHC) staining [14]. IHC is one of the pillars of modern diagnostic pathology and a fundamental research tool in both pathology and translational research laboratories. Conventional IHC staining is a delicate process that requires accurate control of time, temperature, and concentrations of the reagents at each tissue handling step; in fact, IHC stains have up to 30% technical staining failures [15]. Focusing on HER2, a pivotal breast cancer-related protein, we generated virtual HER2 IHC images from the autofluorescence images of unlabeled breast tissue sections, matching the bright-field images captured after the standard IHC-staining while reducing the turnaround time to minutes.

3. Future developments and utilization of virtual histology technology

The ability to virtually stain label-free tissue sections can potentially restructure the clinical workflow in pathology (Table 1). The current virtual staining process takes a few minutes per slide, and this image synthesis and virtual stain creation time could be dramatically shortened by using dedicated hardware and reach real-time performance, which might especially be useful in intra-operative pathology consultation. By applying multiple stains to a single tissue section, alongside the added capabilities of stain blending, synthesis, and micro-structured virtual staining, we will be able to create novel stains that may better highlight cellular structures and organelles. Pathologists will be able to create customized stains that will maximize their morphological diagnostic abilities while minimizing the examination time.

Table 1. Comparison of conventional and virtual tissue staining.

	Conventional staining	Virtual staining
Time to perform	Hours to days	Minutes
Manual labor	Semi-automated process (requires qualified technicians)	Minimal (e.g. paraffin embedding, tissue cutting)
Stain variability	Significant	Minimal
Hazardous waste composition	Tissue fixatives, preservatives and staining dyes	Tissue fixatives and preservatives
Tissue disruption (Destructive)	Yes	No
Multiple stains on a single slide	No	Yes
Stain-to-stain transformation	No	Yes
Technical failures (e.g. IHC)	High	Low

We believe that this deep learning-based virtual staining framework paves the way for new applications in life sciences and biomedical diagnostics. For example, the significantly reduced IHC staining time could help clinicians to provide better patient care, especially in short-staffed pathology departments where IHC can take nearly a week to perform and can result in significant delays in the treatment of patients. Furthermore, this technology could potentially be used to virtually stain cellular elements that current IHC methods fail to highlight, such as heavily masked antigens, proteins with low expression levels and possibly assist in the detection of genomic aberrations (e.g. oncogene amplifications, deletions and fusions) that require expensive ancillary tests not available in many pathology departments.

In summary, virtual staining technology has transformative potential in enabling tissue diagnoses and helping clinicians to provide better patient care while reducing costs, labor and time-to-diagnosis.

Declaration of interest

A.O. has pending patent applications on virtual staining of tissue and is a co-founder of a company (Pictor Labs) that aims to commercialize virtual tissue staining related technologies. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewers disclosure

Peer reviewers on this manuscript have no relevant financial relationships or otherwise to disclose.

Additional information

Funding

Authors acknowledge the US National Science Foundation funding (PI: Aydogan Ozcan).