Detecting entropy increase in categorical data using maximum entropy distribution approximations

ABSTRACT

We propose a statistical monitoring method to detect the increase of entropy in categorical data. First, we propose a distribution estimation method to approximate the probability distribution of the observed categorical data. The problem is formulated as a convex optimization problem, which involves finding the distribution that maximizes Shannon's entropy with the constraint defined by the given confidence intervals on possible distributions. Then we use this procedure to estimate the non-parametric, maximum entropy distribution of an observed data sample and use it for statistical monitoring based on a χ²-test statistic. This monitoring scheme was found to be effective in detecting entropy increases in the observed data based on various numerical studies and a real-world case study.

KEYWORDS:

• Maximum entropy approximation
• statistical process monitoring
• information-theoretic measures
• non-parametric distributions
• χ² test

Funding

This work is supported by National Science Foundation grant #1343969.

Additional information

Notes on contributors

Devashish Das

Devashish Das is a Research Associate in the Health Care Systems Engineering Program at the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery. He received his Ph.D. in Industrial Engineering from the University of Wisconsin–Madison in 2015 and B.Tech. in Manufacturing Science and Engineering from the Indian Institute of Technology–Kharagpur in 2010. His research interests lie at the intersection of statistics, operation research, and systems engineering to improve complex service systems, with a focus on advancing the science of health care delivery. He is a member of IISE and INFORMS.

Shiyu Zhou

Shiyu Zhou is a Professor in the Department of Industrial and Systems Engineering at the University of Wisconsin–Madison. He received B.S. and M.S. degrees in Mechanical Engineering from the University of Science and Technology of China in 1993 and 1996, respectively, and an M.S. in Industrial Engineering and a Ph.D. in Mechanical Engineering from the University of Michigan in 2000. His research interests include in-process quality and productivity improvement methodologies by integrating statistics, system and control theory, and engineering knowledge. He is a recipient of a CAREER Award from the National Science Foundation and the Best Application Paper Award from IISE Transactions. He is a fellow of ASME and a member of IISE, INFORMS, and SME.