Segmentation of cervical nuclei using convolutional neural network for conventional cytology

ABSTRACT

Although implementing the Pap smear has drastically reduced the mortality rates from cervical cancer, false positives and negatives are related to the quality of the analysis and the cytopathologist experience. An alternative is the insertion of digital cytology in the quality monitoring to assist the screening. However, conventional cytology is still a major challenge, as it presents a lot of cellular overlap and several epithelial structures that make it difficult to implement computational methodologies. This article compares the performance of U-net and SegNet neural networks for nuclei segmentation in cervical images. Experiments were performed with different activation functions, batch sizes, and datasets, ISBI (synthetic images from liquid cytology) and CRIC Cervix-Seg (conventional cytology real images). The models achieved a Dice coefficient of 0.9783 for ISBI2015 and 0.9429 for CRIC Cervix-Seg. These results suggest a methodology capable of segmenting real images of cervical nuclei with quality, even in situations of overlap and artefacts, advancing efforts towards the automation of tasks as part of the cytopathological analysis in the laboratory work routine.

KEYWORDS:

• Cervical nuclei
• segmentation
• convolutional neural network
• pap smear

Acknowledgements

The authors would like to thank the Universidade Federal de Ouro Preto (UFOP), the Center for Recognition and Inspection of Cells (CRIC), the Group of Optimization and Algorithms (GOAL), the Microscopy facility of the Biological Sciences Research Center (NUPEB), and the eXtended Reality 4 Good (XR4GoodLab) for supporting this research.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. https://database.cric.com.br/.

2. https://database.cric.com.br/.

3. colab.research.google.com

4. www.tensorflow.org.

Additional information

Funding

This work was supported by the Coordenaç’ão de Aperfeiçoamento de Pessoal de Nível Superior - Brazil (CAPES) under Grant [Finance Code 001]; Fundaç’ão de Amparo à Pesquisa do Estado de Minas Gerais - FAPEMIG under Grants [APQ-00751-19, APQ-01306-22, APQ-01518-21, PPM/CEX/FAPEMIG/676-17, and PPSUS-FAPEMIG/APQ-03740-17]; Conselho Nacional de Desenvolvimento Científico e Tecnológico - CNPq under Grants [303266/2019-8, 305895/2019-2, 308947/2020-7]; Pró-Reitoria de Pesquisa, Pós-Graduaç’ão e Inovaç’ão - PROPPI/UFOP under Grants [19/2020, 23109.000928/2020-33, and 23109.000929/2020-88]; Ministry of Health under grant [905103/2020]; and Moore-Sloan Foundation, and Office of Science, of the U.S. Department of Energy [under the Contract No. DE-AC02-05CH11231]. The authors also acknowledge the Federal University of Ouro Preto (UFOP) and the Coordination for the Improvement of Higher Education Personnel (CAPES) through EDITAL PROPPI/UFOP Nº 18/2022 for the support of the language proofreading service costs.