Reliability as Lindley Information

Abstract

This paper introduces a definition of reliability based on Lindley information, which is the mutual information between an observed measure and latent attribute. This definition reduces to the traditional definition of reliability in the case of normal variables, but can be applied to any joint distribution of observed and latent variables. Importantly, unlike traditional definitions of reliability, this formulation of reliability applies to vector- or matrix-valued estimates and summaries of responses, and therefore generalizes reliability to sets of scores and estimates in addition to individual scores and estimates. This formulation also leads to new bounds for reliability, as well as newly reported relationships between reliability and the traditional Fisher information function familiar in item response theory literature. This form of reliability can be estimated using formulae, or methods used in Bayesian inference such as Markov Chain Monte Carlo (MCMC) depending on the case. Examples based on well-studied datasets are provided, as well as applications to drift-diffusion modeling and randomly-varying intraindividual covariance structures.

Keywords:

• Bayesian statistics
• reliability
• information theory

Article information

Conflict of interest disclosures: The author signed a form for disclosure of potential conflicts of interest. The author did not report any financial or other conflicts of interest in relation to the work described.

Ethical principles: The author affirms having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was not supported.

Role of the funders/sponsors: No funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The author would like to thank William Revelle for his comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the author alone, and endorsement by the author’s institutions is not intended and should not be inferred.

Notes

1 I thank William Revelle for noting the connection to set correlation, as a reviewer.