Automatic estimation of ulcerative colitis severity from endoscopy videos using ordinal multi-instance learning

ABSTRACT

Ulcerative colitis (UC) is an inflammatory bowel disease characterised by inflammation of the large intestine. UC severity is often represented by the Mayo Endoscopic Subscore (MES) which quantifies disease activity from endoscopy videos. In clinical trials, an endoscopy video is assigned an MES based upon the most severe inflammation observed in the video. For this reason, severe inflammation spread throughout the colon will receive the same MES as an otherwise healthy colon with a small localized inflammation. Therefore, the extent of disease activity throughout the colon may not be completely captured by the MES. In this work, we aim to automatically estimate MES for each frame in an endoscopy video to provide a higher resolution assessment of UC. Because annotating severity at the frame-level is highly expensive and labour-intensive, we propose a novel weakly supervised, ordinal classification method to estimate frame severity from video labels alone. Using clinical trial data, we demonstrate that our models achieve substantial agreement with ground truth labels, comparable to the agreement between expert clinicians. These findings indicate that our framework could serve as a foundation for novel clinical endpoints, based on a more localised scoring system, to better evaluate UC drug efficacy in clinical trials.

KEYWORDS:

• IBD
• Ulcerative Colitis
• endoscopy video
• severity estimation
• weak supervision
• multi-instance learning
• ordinal classification
• deep learning
• convolutional neural networks

Acknowledgments

We give special thanks to Dr. Louis Ghanem, Dr. Kathleen Lomax, Dr. Laurie Conklin, and Dr. Susana Gonzalez for providing their GI expertise in scoring a set of endoscopy video frames for use in our method validation and analysis. Institutional review boards and ethics committees approved the protocol for the data used in this study and all patients provided written informed consent. There are no relevant financial or non-financial competing interests to report.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

1. For simplicity, and by clinical convention, we choose to represent these labels with numerical values (i.e. $0<1<2<3$), but the numerical distance between labels need not be equal.

2. As per clinical trial protocol, biopsies were taken with forceps at a predefined location in the left colon for all patients and therefore did not correlate to frames with severe disease.

3. The (unweighted) Fleiss $\kappa$ reported by studies including our own data, are not necessarily comparable to a weighted Cohen’s $\kappa$, so we do not make claims of comparison to human raters at the video-level.

Additional information

Notes on contributors

Evan Schwab

Evan Schwab, Ph.D. is a Senior Data Scientist in Computer Vision at Janssen R&D Data Science. Evan develops advanced deep learning-based computer vision algorithms to analyze large-scale medical imaging and clinical trial data and data science strategies to enhance the clinical research portfolios of Janssen R&D. Evan has worked to develop data science solutions in the therapeutic areas of Immunology, Oncology, Radiology, and Neuroscience. Before joining Janssen, Evan was a deep learning research scientist at Philips in Cambridge, MA.  Evan received his Ph.D. from Johns Hopkins University in Electrical and Computer Engineering where his thesis focused on medical computer vision applied to diffusion MRI. Evan received his B.A. from Cornell University in Mathematics.

Gabriela Oana Cula

Gabriela Oana Cula, PhD is a director of Data Science at Janssen R&D, focused on defining and prioritizing high impact use cases that employ Imaging and Computer Vision in Immunology, with the goal of generating insights, improving endpoints and enhancing clinical development. Prior to joining Janssen, Oana was leading the Diagnostics and Data Science team of J&J Consumer R&D. Oana joined J&J in 2005 as a postdoctoral scientist, after she obtained her Ph.D. in Computer Vision from the Computer Science department of Rutgers, The State University of New Jersey, and she holds a BE in Electrical Engineering from Polytechnical University of Bucharest, in Romania. 

Kristopher Standish

Kristopher Standish is a Director of Data Science at Janssen R&D, focused on the application of artificial intelligence to medical imaging. He leads a team of data scientists who develop and deploy novel solutions incorporating computer vision technologies with the goal of improving the therapeutic development and healthcare delivery. Kris has been with Janssen for 5 years, working with real-world data, medical imaging, and -omics data across multiple disease areas, including Oncology, Immunology, and COVID-19. Prior to Janssen, he earned his PhD from the University of California, San Diego performing research in statistical genetics and bioinformatics, and received a BA from Johns Hopkins studying neuroscience and mathematics.

Stephen S. F. Yip

Stephen Yip is a principal scientist in Data  Science at Janssen Research and Development. He has over 10 years of experience applying advanced image processing and artificial intelligence (AI) tools to structure and analyze large-scale imaging, clinical, and genomic data. Before joining Janssen, he was a Director of Imaging AI at Tempus (Chicago, IL) and later, the Head of Research at AIQ Solutions (Madison, WI). His research focused on the development of imaging-based biomarkers for improving precision medicine and clinical trial design. 

Aleksandar Stojmirovic

Aleksandar Stojmirovic is a Director at Janssen R&D Data Science, leading a team with a mission to manage and organize Janssen’s translational data over the entire therapeutic development cycle and to deliver innovative platforms and solutions that utilize this data to contribute towardseffective and timely portfolio decisions.Before establishing his team, Aleksandar worked as a computational research scientist within Janssen’s Immunology Therapeutic Area, supporting bench scientists to develop hypotheses about disease mechanisms, novel therapeutic targets, biomarkers, and combination therapies using molecular networks derived from human data collected as part of clinical trials. Prior to joining Janssen, Aleksandar was a Visiting Fellow and Research Fellow at the National Center for Biotechnology Information (NCBI) in Bethesda, Maryland, where he mostly focused on developing theoretical frameworks and computational tools for biological network analysis. He earned his Ph.D. in mathematics at Victoria University of Wellington, New Zealand.

Louis Ghanem

Louis Ghanem, M.D., Ph.D. is a physician-scientist and director of translational science and medicine in the immunology therapeutic area at Janssen Research and Development. He is a board-certified pediatric gastroenterologist with expertise in Inflammatory Bowel Disease responsible for driving the strategy and execution of early drug development programs from late lead optimization to proof-of-concept clinical trials. Before joining Janssen, he was an Assistant Professor of Pediatrics at the Children’s Hospital of Philadelphia and Perelman School of Medicine at the University of Pennsylvania.

Christel Chehoud

Christel Chehoud, PhD, is a Director of Data Science, Analytics & Insights at Janssen Pharmaceutical Companies of Johnson & Johnson. In this role, Christel leads a team of data scientists to design, develop, and field impactful data science solutions spanning the entire R&D value chain from discovery through late development. She manages a portfolio of advanced analytics projects, including predictive modeling of disease progression, clinical trial optimization, and biomarker identification. Christel provides technical guidance to conceive and implement Data Science solutions to business questions, interfacing with scientific domain experts to ensure alignment, and leading and advising on collaborations with external Data Science companies. Prior to J&J, Christel was a researcher at the University of Pennsylvania, studying microbial genomics and building analytical pipelines to investigate the human gut virome. Christel holds a Ph.D. from the University of Pennsylvania and a A.B. in Molecular Biology and Computer Science from Princeton University.