Current status and challenges in establishing reference intervals based on real-world data

Abstract

Reference intervals (RIs) are the cornerstone for evaluation of test results in clinical practice and are invaluable in judging patient health and making clinical decisions. Establishing RIs based on clinical laboratory data is a branch of real-world data mining research. Compared to the traditional direct method, this indirect approach is highly practical, widely applicable, and low-cost. Improving the accuracy of RIs requires not only the collection of sufficient data and the use of correct statistical methods, but also proper stratification of heterogeneous subpopulations. This includes the establishment of age-specific RIs and taking into account other characteristics of reference individuals. Although there are many studies on establishing RIs by indirect methods, it is still very difficult for laboratories to select appropriate statistical methods due to the lack of formal guidelines. This review describes the application of real-world data and an approach for establishing indirect reference intervals (iRIs). We summarize the processes for establishing iRIs using real-world data and analyze the principle and applicable scope of the indirect method model in detail. Moreover, we compare different methods for constructing growth curves to establish age-specific RIs, in hopes of providing laboratories with a reference for establishing specific iRIs and giving new insight into clinical laboratory RI research. (201 words)

Keywords:

• Reference interval
• real-world data
• indirect method
• data mining
• specificity

Introduction

Real-world data (RWD) and real-world evidence (RWE) have long been applied in medicine, such as in the discovery in the eighteenth century that citrus fruits can be used to treat scurvy. RWD include routine data collected from various sources related to patient health status or healthcare provision for patients, in addition to traditional clinical trials [1]. Alongside medical data such as those derived from hospital information systems (HISs), laboratory information systems (LISs), and health information management systems for physical examination, health data derived from personal wearable devices and personal health applications are also valuable. In recent years, with the wide application and development of electronic medical records (EMRs), real-world research has again attracted attention. U.S. Food and Drug Administration statements have been successively published, and guidelines have been proposed around the use of RWD and RWE in review and regulatory processes, such as those of pharmaceutical and biological products [2–4], bringing real-world research into focus.

The rapid development of medical information technology and big data in medicine has opened many possibilities for the application of RWD, such as the regulation of medical devices, the development of safe and effective treatments for drugs, delivery of patient care, research in healthcare systems, and establishment of disease diagnosis or prognostic models [5,6]. RWD can also be used in clinical laboratories to establish reference intervals (RIs), real-time quality monitoring systems [7], and automatic audit systems for test results, thereby greatly improving the efficiency of clinical laboratory testing, diagnostics, and treatment.

RIs refer to intervals between the upper and lower reference limits [8] and are usually taken as the reference value of healthy individuals in the middle 95% (i.e. between the 2.5% and 97.5% quantiles) of the reference range. RIs are useful for judging patient health status and making clinical decisions such as diagnosis and treatment [9]. In routine work, most RIs used in laboratories are cited from documents such as industry guidelines, textbooks, and instrument or reagent manufacturers’ instructions. However, the large number of laboratories worldwide, wide variety of instruments and reagents used range of detection methods available, and application of international standards all highlight the need for validation of RIs [10]. Due to differences in population, geography, lifestyle, and epidemiological manifestations of disease, biological RIs also vary significantly between individuals [11], and RIs directly imported from documents may not be applicable to local populations. Therefore, for many test indicators, it is often necessary for different laboratories to establish specific RIs to partition patients based on sex, age, population, ethnicity, and so on.

A document titled “Defining, Establishing, and Verifying Reference Intervals in the Clinical Laboratory: Approved Guideline–Third Edition (EP28-A3c)” [8] put forth by the Clinical Laboratory Standards Institute (CLSI) and the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) in 2010, and the guidelines issued by the Société Française de Biologie Clinique (SFBC) [12], recommend the “direct method” as the gold standard for establishing RIs by recruiting healthy individuals of adequate sample size (n ≥ 120). Although RIs established via this traditional method are relatively reliable, most laboratories find it difficult to directly recruit a sufficient number of “healthy” reference individuals. In addition, it is difficult to judge the definition of “health” for specific groups such as newborns and the elderly, and there are ethical limitations that make it more difficult to recruit adequate samples from such populations [13]. With the development of real-world research, many scholars have begun to use the method of data mining on RWD to establish indirect reference intervals (iRIs) [14]. RWD, derived from real medical environments, reflect the actual health status of patients and is widely used in clinical laboratories. Data mining methods are economical and practical, fit realistic data, have strong extrapolation capabilities, and can help establish population-specific RIs for children, older adults, and pregnant women. However, there are also some problems with their application. Many factors affect data quality in clinical laboratories, such as preanalytical factors and analytical factors, and the stability of iRIs is difficult to guarantee. Limited access rights to EMRs and incomplete clinical information, such that exclusion criteria cannot be strictly enforced, may lead to non-negligible selection bias. In addition, it is difficult to guarantee the security of data and the privacy of patients when using LIS data [15,16]. Data analysis is also challenging for physicians in district and community hospitals, who are particularly strained on weekdays and may have limited time to learn complex statistical methods for establishing iRIs using RWD.

At this stage, the establishment of iRIs is characterized by problems such as non-uniformity in steps and the numerous algorithms of modern models without any official guidance on the selection of a suitable algorithm. Although a multitude of studies has been done on indirect methods for establishing RIs, the accuracy of their findings needs verification, owing to a lack of uniform standards. To resolve the aforementioned problems, this paper will summarize several general steps for establishing iRIs based on RWD: data extraction, data cleaning, describing data distributions, statistical processing, validation, and publication of iRIs. Furthermore, we provide a brief overview of various statistical methods and their advantages and disadvantages, outline the current state of research on establishing iRIs based on RWD, and provide a reference for laboratory workers on the subject as well.

Methods of establishing RIs

There are two methods for establishing RIs: (1) the conventional method of direct sampling and (2) the big data method of determining RIs based on RWD for retrospective analysis using appropriate statistical methods, also called the indirect sampling method.

There are both a priori and a posteriori processes for establishing RIs. The a priori approach aims at including reference individuals who meet the criteria after setting adequate inclusion criteria, which is suitable for tests that have mature detection procedures. The a posteriori approach aims at excluding group samples after they have been drawn, tested, and analyzed, and is appropriate for new tests or less-developed test procedures. The direct method is commonly used as an a priori approach, with steps generally including the collation of known sources of variation, the development of a questionnaire, the establishment of strict inclusion/exclusion criteria, assessment of the fitness levels of reference individuals, and screening a sufficient number of reference individuals that meet all the above criteria.

In general, direct methods have the advantages of accurate results and adequate representation of study subjects when the preanalytical, analytical, and post-analytical conditions are well controlled. However, because “health” is difficult to define and the selection of reference individuals is the most difficult aspect of the direct approach, the selected screening criteria and the presence of subclinical disease states will directly influence the values of the biological RIs obtained. In addition, recruitment of sufficient reference individuals is inherently difficult for general laboratories, owing to which the EP28-A3c document allows the establishment of iRIs.

Indirect methods can be divided into two categories. The first involves establishing iRIs by obtaining data that approximate the distribution of the community population, such as those derived from physical examination and blood donation centers. The second category involves establishing iRIs using “mixed distribution” data stored in the laboratory, which are statistically processed to isolate “normally distributed” data. Methods and processes for determining iRIs have been developed for more than 50 years, and medical big data analysis, with real and adequate data sources, is an important aspect of RWD research that provides convenience for RI research, as well as providing new schemes for various aspects such as laboratory process optimization and quality control [17].

Studies of iRIs based on RWD have several advantages over traditional methods [9,18–21]. The most prominent advantage is the large volume of available data. The sample size of the database often remains adequate after excluding samples that do not meet the requirements, favoring the study of variant sources, enabling the comprehensive analysis of relevant stratification, and even establishing RIs for rare or expensive detection indicators. Second, the data stored in the laboratory are obtained from a mixed distribution of “healthy” individuals versus “non-healthy” individuals, rather than “apparently healthy” individuals specifically recruited after rigorous screening, which fits the population distribution and therefore strongly represents the population served by RIs in reality. Third, the indirect method requires no reference population recruitment, sample collection, and other related steps, which significantly reduces the workload of researchers, improves the efficiency of the study, and greatly reduces costs. Fourth, it is advantageous to mine and compare the same detection indicators from multiple laboratories, facilitating better coordination and comparison between laboratories and promoting the establishment of multicenter RIs within countries or regions [22]. In addition, indirect methods have ethical advantages, avoiding invasive specimen collection for reference individuals, and also avoiding similar less ethically compliant events, particularly vulnerable groups such as infants.

Establishment of iRIs from real-world data

The general process of the indirect method has been summarized in several previous publications [19,23,24]. It is typically divided into the following steps: data extraction, data cleaning, data distribution description, statistical processing, RI verification, and publication. The specificities of the process are shown in Figure 1.

Figure 1. Flowchart for establishing reference intervals from real-world data. SD: Standard Deviation; LAVE: Latent abnormal values exclusion; LIMIT: Laboratory Information Mining for Individualized Thresholds; RIs: Reference intervals; Cis: Confidence intervals.

Sample size

The sample size for the direct method recommended by the CLSI Working Group guidelines is at least 120, which is not an optimal number for the indirect method. The sample size of RWD required can be affected by the skewed distributions of reference populations, the proportion of pathological data, and the degree of overlap between populations [25]. Larger sample sizes are required to establish iRIs for partitioning with more variability [26]. In recent years, some scholars have explored the topic of sample size needed via research and have proposed sample size procedures and calculation algorithms [27–29]. Linnet [30,31] proposed methods for estimating sample size, and indicated how sample size is related to data distribution and whether a larger sample size is required when the distribution is clearly skewed. Ammer et al. also put forward the same opinion, using Z-score deviation to evaluate the accuracies of established iRIs when the sample sizes were 1000, 5000, 50,000, and 500,000, and authors stated that the sample size of over 5000 was required when there was large disease bias in RWD [32].

Data preprocessing

Unlike direct methods, which have strict inclusion and exclusion criteria, the volume of data included in indirect methods can be huge, and the processing of the data is critical to the results of the iRIs, in addition to guaranteeing the quality of the data (controlling preanalytical and analytical factors). Theoretically, the results of routine tests in healthy individuals all exhibit a Gaussian or near-Gaussian distribution [33], while skewed distributions arise mostly from both disease factors and physiological factors [25,32,34], and improper transformation of skewed data can produce erroneous RIs. Therefore, adequate preparation is necessary before the establishment of iRIs. Based on the observation or review of relevant literature, the initial assessment can reveal relevant diseases and physiological factors such as sex, weight, and age that may affect the level of indicators, thereby preliminary exclusion of abnormal data is carried out by screening diagnostic, departmental, and other information for establishing iRIs. Subgroups with clear differences are partitioned, overlaps between data are removed, and then the difficulty of obtaining a “non-disease” distribution is resolved using a statistical approach for the classification of clinical information [23]. In addition, data from patients with repeated measurements should also be screened, keeping data from only the first or last test, as patients with multiple laboratory tests are more likely to be in a diseased state [35].

Data description

Preprocessed data distributions such as the near-Gaussian distribution are not satisfactory, and relatively suitable methods can also be applied to transform or describe the data with other distributions, such as the Bhattacharya method applied to Gamma distributions. Log transformation is the most commonly used method and works well to adjust data to a normal distribution when the skewedness of the analyte distribution is explained by available biological factors [33]. The modified Box-Cox transformation [36,37] is also commonly used and is considered a more flexible transformation method [38]. However, there is often a risk of incorporating prevalence data this way, leading to the inadvertent widening of RIs [39,40]. Other non-Gaussian transformation methods such as Manly transformation and Erlang distribution [20] are also available, but care must be taken when performing data transformation to avoid over-transformation, which can cause the RI to lose specificity [23]. Latent class distribution regression [41] is the most effective method to use when a disease causes a large bias, for example, but applying it when a disease causes less bias may overestimate the reference limit.

Elimination of outliers

Removal of outliers is an important step in obtaining an “approximately healthy” distribution, and there have been a number of publications summarizing different ways to remove them [8,19,20,23,38]. These include the Dixon method [42] (for nonparametric methods), the Tukey robust method [43], the Horn method [36] (performing a Box-Cox transformation and then applying the Tukey robust method), the Laboratory Information Mining for Individualized Thresholds (LIMIT) method for identifying outlier-related diagnostic codes by unsupervised machine learning [44], the Latent Abnormal Value Exclusion (LAVE) method [38], and others. Some scholars have applied different outlier removal methods to run comparative analyses using cohorts of data [45,46]. Tan et al. [45] found that the kosmic algorithm performed best by comparing eight methods for excluding outliers or identifying pathological distributions.

Partitioning of subgroups

Whether direct or indirect, laboratories should generally establish different RIs for the same test item based on populations with different characteristics [9,20,47–49]. Many scholars have made suggestions on how to best partition subgroups when establishing RIs, and the EP28-A3c document discusses several possible partitioning criteria such as sex, ethnicity, age, gestational period, and race. In addition, differences in blood groups, genetic constitutions, menstrual cycles, lifestyle habits, economic statuses, geographical environments [50,51], diurnal variations, and climatic seasons [52–54] may also contribute to different normal levels of various indicators [34]. Statistical difference analysis is also needed to judge the necessity of dividing subgroups according to literature or expert recommendations. The main statistical methods for this are the Harris Boyd method [55,56], Lahti method [57], Fraser method [58], and Ichihara method based on a 3-level nested ANOVA [51]. Ichihara et al. published a comprehensive summary of these approaches in a previous report [38].

Age-based partitioning

To improve the sensitivity and specificity of RIs, different subclasses need to first be defined, based on the characteristics of the reference individual [9,34], with the most common partition criteria being sex and age, which most recent studies take into account [59–64]. However, stratification by age is more difficult than stratification by sex. In infants, particularly newborns, existing RIs do not necessarily correctly account for health status, as seen in studies aimed at screening newborns for congenital hypothyroidism [65,66], which have increased false-positive rates as well as unnecessary follow-up due to the absence of an age-stratified thyrotropin RI. In addition, children and adolescents differ greatly from adults in terms of growth, development, and dietary habits [13], and changes in some specific biomarkers are associated with normal physiological growth and development (e.g. creatinine and alkaline phosphatase) [64,67–69]. All of these factors highlight the need to establish age-specific RIs [70]. Menopause is an important age when women progress to a new physiological state, where the gradual decline of ovarian function leads to changes in the levels of indicators such as hormones [34,71]. A few studies have established relevant RIs around menopause [72–74], but it is clear this physiological stage has not received enough attention. In addition, relevant studies of RIs in the elderly are fewer and more challenging. The main difficulty surrounding them has to do with the selection of healthy individuals; many elderly people have chronic diseases and need to take various prescription medications. In addition, many elderly people use health products and Chinese patent medicines. These factors make it difficult to distinguish variation among laboratory values due to physiological ageing versus morbid states and drug effects. During the twenty-first century, the proportion of the global elderly population has been increasing year over year [75], and more attention has been paid to the health statuses of the elderly [76]. For example, studies conducted by Eggers et al. suggested that N-terminal pro-B-type natriuretic peptide (NT-proBNP) and troponin T levels change with age in the elderly subgroup [77]. More relevant studies have aimed at establishing geriatric-specific RIs in recent years [59,60,78–80]. However, the CLSI guideline Defining, Establishing, and Verifying Reference Intervals [8] is not entirely applicable to the elderly population. Thus, more targeted and guided statistical methods based on the special characteristics of the elderly population are needed. Despite these difficulties, it is still necessary for laboratories to establish specific RIs for specific stages of life.

Ethnic and regional partitioning

In addition to well-known stratifications by sex and age, ethnicity and region are other potential factors for stratification [50,51,81]. Many studies have shown the influence of racial factors on reference ranges [82–87]; however, most RI studies do not stratify according to racial factors. Most countries are ethnically heterogeneous, and individuals of different ethnicity may exhibit differences in reference values of some indicators due to genetic factors, lifestyle habits, and others. Standard reference individuals for existing RIs are mostly European/American Caucasians, although other ethnic populations form the worldwide majority. For instance, the median creatine kinase activity of healthy Black men is approximately twice that of Caucasians [88]; thus, RI studies must take ethnicity into account. In addition, different countries and regions often have varied geographical characteristics and different topographies can also affect RIs; for example, the average hemoglobin level of people living in high-altitude areas is higher than that of people in other areas. Therefore, the establishment of RIs must be tailored, and the use of RIs that do not match the characteristics of the local population may lead to misinterpretation of test results.

Gestational partitioning

Women experience their own characteristic physiological changes during and after pregnancy. The first trimester in particular is an important period for predicting pregnancy complications or adverse pregnancy outcomes, and it is important to closely monitor every indicator during this period [89]. The majority of current RIs have been established based on non-pregnant healthy women as reference individuals, and whether they are suitable for use in pregnant women needs to be evaluated [90]. For example, RIs of thyroid function indicators during pregnancy have attracted much attention from scholars worldwide [91–93]. According to large epidemiological studies, thyroid disorders in pregnancy are associated with severe maternal, fetal, and neonatal complications. Therefore, it is crucial to accurately diagnose thyroid disease in pregnancy at an early stage, which implies the need to ensure the availability of precise RIs for thyroid function indicators specific to pregnant women.

With the continuous progress of laboratory RI research, subgroup partitioning should be taken into account in most studies when establishing RIs. However, it remains controversial whether these partitions are necessary as RIs become increasingly more refined with finer partitioning. How to determine whether the grouping criteria are correct remains a challenge. While some scholars have illustrated the importance of proper stratification [11,94], others have proposed new solutions [95]. For example, health states can be more effectively assessed by longitudinally comparing personalized reference values for an individual patient and assessing changes over time. With the development of precision medicine and the maturation of big data-related technologies, this is perhaps the future trend of RIs [21].

Validation of RIs

Newly established RIs need to be validated before use [8,9,20,96]. Briefly, validation can be performed by collecting a small number of homogeneous healthy reference individuals (n = 20 or 60) from a laboratory that fulfills the exclusion criteria and partitioning requirements, removing the outlier results, and replenishing the eligible reference individuals, then repeating these steps to obtain 20 or 60 results without outliers. If no more than 10% of the results fall outside the established RI, the RI can be used.

Validation is also necessary when laboratories are going to use literature-derived RIs, guideline-derived RIs, or RIs established by other laboratories. If all study conditions are met, validation of proposed RIs is performed by checking the consistency of the sample collection process, the analysis system, and the population characteristics of the subjects. After validation as per the above method, the proposed RIs can be put into use.

Selection of statistical methods

Statistical models for establishing iRIs

Statistical methods for the official establishment of RIs recommended by the EP28-A3c document [8] include parametric, nonparametric, and robust methods. If the data conform to a normal distribution or are transformed to arrive at a normal distribution, a parametric method can be used to calculate confidence intervals (usually 95% confidence intervals, as the mean ± 2 SD) for normally distributed datasets as RIs. If the data are also transformed to arrive at a normal distribution and the sample size is ≥120 patients for each subgroup, a nonparametric approach is generally used, taking values between the 2.5th and 97.5th percentiles of the data distribution as the RI. An exception to this is special markers, such as cardiac troponin, for which the 99th percentile of the data distribution should be selected as the upper reference limit based on required guidelines [97]. The parametric and non-parametric methods are also frequently used when establishing RIs via direct methods; applying them in indirect methods requires clear identification and handling of outliers. The robust method is an iterative algorithm for estimating RIs with small sample sizes and is applied in studies with sample sizes of <120 subjects. However, there is still a need to ensure that the data are symmetrically distributed. This method is also applicable to big data analysis and requires the use of bootstrap methods to calculate confidence intervals for RI bounds. A previous study [98] contrasted these three methods, stating that appropriate methods should be selected based on sample size and data inclination, and the performances of the three methods are influenced only by these two factors.

In recent years, many scholars have proposed the need to establish new algorithms based on iRIs of mixed distribution data or created related computerized instructions, such as the reference value advisor instructions for small sample groups [99]. In this vein, there are ongoing efforts to obtain distributed data that are “approximately healthy”. A summary of the indirect modeling methods are shown in Supplementary Table 1.

Hoffmann method

The Hoffmann method, proposed in 1963, is an algorithm for plotting scatterplots of cumulative frequencies on normal probability paper. In this graphical algorithm, data from mixed distributions are presented in different linear regions to identify “diseased individuals,” thereby using non-diseased datasets to establish iRIs. Several variants of the Hoffmann method subsequently emerged, among which was an improvement proposed by Katayev et al. [100], which quickly came into question [101–103]. Some researchers claimed that this variant went against the basic theory underlying the Hoffmann method and caused the established iRIs to be too narrow. Holmes et al. evaluated the method proposed by Katayev et al. along with the correct Hoffmann method, as well as the Bhattacharya method, and suggested alternative methods suitable for use in modern times [102].

Bhattacharya method

The Bhattacharya method is a graphical method that appeared in the literature after the Hoffmann method and was subsequently applied in the field of medicine [104]. At first, the method required that the obtained health distribution be normal and later resolved some technical shortcomings after several improvements. A further improvement of this method involves a description of the patient data as having a gamma distribution [105], which can be done using software and computer programs [106], as has been widely applied in many studies [107]. However, many elements in the graphical approach of the Bhattacharya method require a subjective decision to be made by the researcher; thus, the results may not be objective enough.

The Hoffmann and Bhattacharya methods separate distributions using graphical methods, and although intuitive, do not perform well enough. Both methods contain visually identified steps that are prone to human error. In addition to the above two methods, the Pryce method [19,108], which assumes that the central components of the distribution represent observations of healthy individuals and follow a Gaussian distribution, requires clarity as to whether test results for a reference population in a mixed distribution are distributed at one or both ends. Since the method was popularized before computers became commonplace, it is not widely applied today. Nowadays, with the progressive maturation of computer algorithms and rapid advances in information technology, there is a gradual shift of focus to more sophisticated but effective modern methods of mixture model decomposition.

Expectation–maximization algorithm

The expectation–maximization (EM) algorithm [31], an improved method based on the Bhattacharya method, can be applied to Gaussian mixture models. It is an iterative algorithm with two main steps: taking the expectation expression and maximizing the expectation. Its execution starts with an initial round of calculation, with each round of calculation repeating the aforementioned two steps until the various parameters converge and finally cluster into a dataset with different distribution parameters. A head distribution set and a tail distribution set are obtained, combined with the actual selection of the head or tail representative of a healthy distribution, and the separated normally distributed data are run to establish iRIs using a nonparametric method.

Truncated minimum chi-square

The truncated minimum chi-square (TMC) model [109] is a mixture model of discrete data, which assumes that a majority of the dataset consists of “non-morbid” values. It has an emphasis on describing centrally distributed data, and only truncated datasets are used subsequently. This model does not explicitly require the type of morbid distribution, and the “non-diseased” distribution of the data needs to satisfy a power normal distribution. This method can also automate stratified analyses by sex and age and is available in the TMC script in R (a commonly used free open-source programming language for statistics and graphics) [110].

Truncated maximum likelihood estimation model and kosmic model

The truncated maximum likelihood (TML) model was developed by Arzideh et al. [111] and can estimate skewed distributions. This method estimates the so-called non-morbid distribution utilizing a truncated interval for the range of test results in the Box–Cox transformation of the input dataset and a power distribution estimated by the maximum likelihood method for the truncated data (μ, δ, and λ parameters). iRIs are calculated using a nonparametric method. Ahmed et al. [112,113] used the TML method to construct iRIs for alkaline phosphatase and creatinine levels among Pakistani children. Later, Zierk et al. made improvements to the TML method, called the kosmic [114] model. The accuracy of the iRIs calculated via the kosmic model is assessed in the text using simulated datasets and patient samples, which show reliable results with increasing proportions of abnormal test results in <20% of cases with abnormal test results, even in challenging simulation scenarios. The kosmic algorithm has been used in the Next Generation Pediatric RIs project [22]. However, the method requires two steps to establish a continuous RI, beginning by establishing iRIs for discrete age groups and then fuzing the data to derive continuous iRIs.

RefineR model

The refineR model was developed by Ammer et al. [115] in 2021 and employs an inverse modeling approach to improve the quality of the repeat interval estimates as a refinement algorithm. Using simulated datasets based on conventional data and patient samples in the literature, Ammer et al. compared analyses with the kosmic algorithm and the direct method and eventually found that the refined model performed better than the kosmic algorithm for simulated distributions with a high proportion of non-morbid values (>20%), with even better values than the direct method published in simulation studies. Moosmann et al. [116] used this method in 2022 to establish iRIs for neutrophil-to-lymphocyte ratios, platelet-to-lymphocyte ratios, and lymphocyte-to-monocyte ratios.

Self-organizing map model

Yan et al. [117] proposed an application of the self-organizing map (SOM) model to the calculation of iRIs. In their report, iRIs established using the SOM model were compared to those established using the mixed Gaussian distribution model (EM algorithm). The iRIs estimated via the mixed Gaussian distribution model were considered to be consistent and stable but not for skewed distributions. While the SOM model is suitable for high-dimensional data and data that fit well within the set’s distribution, it is easily affected by sample size and outlier detection strategies.

The principles underlying this algorithmic model are also complex and, for non-specialists, difficult to operate and understand. However, some algorithmic models are implemented with fixed coding, and with the popularity of programming knowledge, most researchers in the future will likely be expected to use these improved new algorithms in combination with machine learning models to assist in the establishment of iRIs. Overall, the new algorithms detailed above are effective at correcting the data distributions of samples containing morbid states and stratifying RIs by clinically relevant covariates, which largely guarantees the accuracy of each study stage. However, there are still many aspects that need to be refined, such as the significant difficulty that lies in evaluating the efficacy of different indirect methods. Ammer et al. [32] have recently developed the “RIbench” software (open-source R package) and proposed a significant novel approach for evaluating indirect methods. Using the absolute z-score deviation, they evaluated five indirect methods by comparing different sample sizes, locations of pathological distribution, degrees of overlap, and scores in the dataset of pathological data proportion. The “RIbench” software can objectively assess the robustness of different indirect methods and is an important tool for exploring more well-established indirect method sampling techniques [25].

Methods of constructing growth curves for establishing age-specific RIs

Age is a continuous variable and does not have obvious cutoff points, making it difficult to divide age intervals for subgroups. For children and adolescents, in particular, the division of age reflects physiological conditions at different developmental periods, and discrete age groupings poorly reflect the dynamic changes in biomarkers with age. This problem is well addressed by plotting continuous growth charts that are currently widely used to establish age-specific RIs in the pediatric population [118].

The easiest approaches to constructing growth curves are the deviation graph method and the percentile method; however, they yield suboptimal smoothness. A number of more advanced methods for fitting growth curves are now available, such as the method used by Mokhtar [119], who applied the restricted cubic splines model to construct continuous RIs for thyrotropin levels. The fractional polynomial theory-based method proposed by Royston [120] showed that relatively conventional polynomials are more flexible in fitting models and can accomplish model fitting with fewer terms. In addition to this, the lambda-median-sigma (LMS) method [121] is an accepted standard approach to fitting growth curves among children and can be used to fit growth curves of height, weight, and body mass index [122]. The generalized additive model for location, scale, and shape [123] adds shape parameters based on the LMS model. It imposes no restrictions on the model distribution, can fit a smoother curve, is more stable and flexible relative to other methods, has become increasingly popular in recent years, and is also highly recommended by the World Health Organization [124]. We summarized the advantages and disadvantages of the above methods for constructing growth curves in Table 1.

Table 1. Comparison of methods for fitting growth curves when establishing age-specific reference intervals.

Methodology	Method of smoothing	Advantages	Shortcomings
Curve method		Simple and intuitive.	Inaccurate evaluation for skewed data. The constructed curve is not smooth enough.
Percentile method [125,126]		Intuitive. Suitable for dynamic observations or group comparisons, distinguishable from within group dispersal.	Cannot overcome skewed distributions. The curve swings are large at both ends.
Cubic splines [127]	Based on polynomial segment fitting.	Takes both smoothness and goodness-of-fit into account.	The smoothing is poor when the percentile curve is not smooth enough.
Kernel estimation [128–130]	Locally weighted average.	Facilitates calculation.	Improper choice of bandwidth can produce significant bias. The existence of boundary effects.
Fractional polynomials [120,131,132]	Fractional polynomial function, maximum likelihood method.	Relatively conventional polynomials are more flexible in fitting models. Model fitting can be done with fewer terms. Fast fitting.	The fitted curve is smooth; however, the effect of covariates is ignored.
Locally weighted scatterplot smoothing [133,134]	Locally weighted regression.	The fitted curves fit the overall trend. Can smooth the data well.	The curve smoothness is related to the proportion of data selected. Inefficient for large datasets.
Radial smoothing technique [135]	Radial basis function.	The underlying distribution needs to be close to normal.	Under the assumption of an approximately normal distribution, the stability is strong and the deviation is small.
Lambda-median-sigma method [121,122]	A smooth curve is drawn using skewness coefficient, median, and coefficient of variation.	The results are intuitive and easy to interpret. Improved comparability of differences between groups. High accuracy.	Applicable for situations with only one explanatory variable..
Generalized additive model for location, scale, and shape [123,136]	Shape parameters are added based on the LMS method, maximum penalized likelihood function.	Allows for parametric modelling of all distributions. Multiple independent variables may be considered in relation to the dependent variable. The fitted curve is smoother.	The larger sample size is required.

Current challenges and prospects

Although indirect methods have substantial advantages in establishing biological RIs, many questions remain unanswered. Among the most important issues are statistical methods and computer technology. Data mining techniques can be unfamiliar and complex to most laboratory staff. There are more and more emerging statistical models for establishing iRIs, and the indirect model is not perfect at present. This makes it very difficult to ensure the accuracy of data analysis for iRIs establishment conducted by clinical laboratories. Moreover, the informatization of EMRs is a great challenge for laboratory personnel. Many laboratory personnel do not have sufficient access to EMRs, and even in laboratories where EMRs are accessible, issues with system compatibility between EMRs and LISs can lead to a series of failures in performing data analysis, such as missing information and formatting errors [137]. Data mining, the foundation for establishing iRIs, relies on filtering access through existing health information to approximate healthy population distributions. Incomplete or inaccurate EMR data can therefore lead to inefficient data cleaning, stalling of studies, and erroneous results. Some researchers have proposed that data warehousing can solve the problem of data access [138]; however, there are security and ethical issues to be considered in data sharing. Data security and privacy breaches, which are prevalent in medical big data research, have attracted much attention, and some studies have proposed effective solutions for this issue [15,16,139]. As the standardization and regulations pertaining to big data improve, data security and privacy can also be better protected.

Data quality issues are also being highlighted lately. It is easy to ignore the influence of preanalytical and analytical factors using indirect methods. The review mechanism for missing criteria related to the extraction process of data, the large amount of data itself, and the improvement of data mining techniques cannot fully compensate for deficiencies in data quality, making it necessary to strictly specify all strategies to improve data quality. Data mining techniques can be strange and complex to most laboratory staff, making it very difficult to guarantee accurate data analysis, and building statistical models of iRIs in clinical laboratories requires advanced training. Information technology has developed rapidly in recent years, and the application of artificial intelligence and machine learning is expected to become key to solving these problems. If EMRs and LISs are upgraded, machines can automatically record and audit relevant variants during specimen collection, transportation, storage, and testing. While the laboratory is continuously monitored for quality using big data technology, the quality of data in the database can be effectively improved. ​Moreover, sophisticated statistical models can simplify the steps that require manual operation through machine learning, thus making it convenient for basic staff to also establish RIs at their own laboratories. With the realization of smarter laboratories, where all steps will be standardized, the comparability of results between different testing institutions will hopefully no longer be a problem that plagues physicians.

Existing research metadata and statistical methods used are mostly opaque, and the data used in a study may have a non-Gaussian distribution. The decision to process the data depends on the investigator, making it highly subjective. Although current guidelines for establishing RIs via RWD mining can cite the CLSI guidelines, the guidelines provide only limited recommendations and are in need of regular updates. Fortunately, many expert scholars have published instructive articles [14,19,20], that detail relevant statistical techniques. The IFCC and CLSI encourage clinical laboratories to establish a consensus on indirect methods for the establishment of RIs, and future standardization will hopefully also improve transparency in RI research.

With the application of computer technology and artificial intelligence to RWD, real-world research is gradually maturing. Making systematic automatic recording and review of relevant variants during laboratory specimen processing a reality, improving the information acquisition module of EMRs, and enabling information sharing with various systems will lead to great improvements in the integrity, accuracy, traceability, and safety of laboratory data. Applying machine learning to build statistical models of iRIs will make the determination process of iRIs easier. With the construction of intelligent laboratories in which all links will be standardized, the mutual recognition of results between different laboratories will no longer be an issue. In addition, the privacy rights of patients will be paid more and more attention to, the relevant big data regulations will be gradually improved, and computer technology can be applied to upgrade database security systems to adequate levels.

Limitations

There are some noteworthy limitations to this review. We were not able to obtain all the articles of the relevant studies mentioned due to problems inherent to the retrieval strategy or to obtain permissions for the necessary literature databases. Moreover, most of the retrieved articles are in English, and RI research articles written in other languages are difficult to include, resulting in a bias against some non-English-speaking countries’ or regions’ findings. We hope to improve our retrieval strategy to solve these issues in the future.

Geolocation information

Shengjing Hospital of China Medical University, No.36 Sanhao Street, Heping District Shenyang, Liaoning Zip, China

Abbreviations
ANOVA	=	Analysis of variance
CI	=	Confidence interval
CLSI	=	Clinical Laboratory Standards Institute
EM	=	Expectation–maximization
EMR	=	Electronic medical record
GAMLSS	=	Generalized additive models for location, scale, and shape
HIS	=	Hospital information system
IFCC	=	International Federation of Clinical Chemistry and Laboratory Medicine
Iri	=	Indirect reference interval
LAVE	=	Latent Abnormal Values Exclusion
LIMIT	=	Laboratory Information Mining for Individualized Thresholds
LIS	=	Laboratory information system
LMS	=	Lambda-median-sigma
NT-proBNP	=	N-terminal pro-B-type natriuretic peptide
RI	=	Reference interval
RWD	=	Real-world data
RWE	=	Real-world evidence
SD	=	Standard deviation
SFBC	=	Société Française de Biologie Clinique
SOM	=	Self-organizing map
TMC	=	Truncated minimum chi-square
TML	=	Truncated maximum likelihood

Disclosure statement

No potential conflicts of interest have been reported by the authors.

Data availability statement

No data, models, or code were generated or used during this study.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by the Natural Science Foundation of Liaoning Province (no. 2021-MS-03) and the “345 Talent Project” of Shengjing Hospital of China Medical University.