The Big Data Paradox in Clinical Practice

Figures & data

Figure 1. The bias-variance trade-off in randomized controlled trials (RCTs). The trial and model are designed based on prior knowledge, which informs the choice of the statistical test to be used, the variables to be included in the model, and whether or not to focus on parsimony or account for anticipated patient heterogeneity. For Bayesian models, prior probabilities are also needed. Once the RCT is activated, it may enroll a diverse patient population. The final data may then be analyzed using either a parsimonious (A) or a more complex (B) statistical model that accounts for patient heterogeneity. (A) In the first scenario, all of the enrolled patients are used to estimate one parameter for the relative treatment effect measured at the HR scale. The estimated HR will have higher bias (higher systematic error) because it assumes the heterogeneous enrolled patient population to be homogeneous, but lower variance (lower random error) because it uses every enrolled patient. (B) In the second scenario, the more complex model generates multiple HRs that take into account the diverse subgroups of patients enrolled, thus resulting in lower bias (lower systematic error). However, each estimated HR is based on a lower effective sample size, resulting in higher variance (higher random error). Figure adapted from images created with BioRender.com.

Table 1. Glossary of key terms.

Term	Description	Related terms	Description
Terrain	The objective underlying reality	Objective truth True parameter value Related terms Description	The objective truth is always inaccessible to our subjective experience The true value of a parameter, such as a hazard ratio, that we are interested in
Map	Models of the terrain we construct to navigate the world	Statistical model	Models of the true parameter value we construct to inform our clinical inferences
Bias	The distance between the true parameter value and the expected value of a statistical model that occurs due to constraints and other assumptions imposed by the statistical model	Systematic error Patient relevance	Bias represents the systematic error in our statistical model Statistical models with high bias are less relevant to individual patients. Answers the question: “how relevant is the statistical model’s estimated parameter to a patient I am seeing in clinic?”
Variance	How close are our measurements to each other	Random error Robustness Standard error Confidence interval	Variance measures random error Robustness is the opposite of variance. Statistical models with high variance are less robust Standard error is the square root of the variance Confidence intervals are functions of the standard error
Reducible error	A combination of the random error due to variance and the systematic error due to bias. The big data paradox occurs because traditional error intervals are functions of the standard error, not the reducible error.	Mean squared error (MSE) Root-mean-square error (RMSE)	The MSE measures the reducible error. Its relationship to the bias and variance of a statistical model is expressed by the equation: $\mathrm{MSE}={(\mathrm{bias})}^2+\mathrm{variance}$ The RMSE also measures the reducible error but is estimated using the same units as the parameter of interest and can thus be used to generate intuitive error intervals for the reducible error. Its relationship to the systematic and standard errors of a statistical model is expressed by the equation: ${\mathrm{ }(\mathrm{RMSE})}^2={(\text{systematic error})}^2+{(\text{standard error})}^2$
Irreducible error	The distance between the true parameter value and the expected value of a statistical model that occurs due to unknown elements that are not represented within the dataset.	Total error	The total error when estimating a parameter is the sum of the irreducible and the reducible error. Thus, even if the reducible error is zero, there will still exist some degree of irreducible error that cannot be quantified within the dataset.

Figure 2. The big data paradox. (A) Our target is the underlying true parameter, but its complexity can never be perfectly modeled (B) Therefore, there will always be some distance between our models of the true parameter and the true parameter itself. This distance is the sum of the bias (blue line) due to constraints and other assumptions imposed by the statistical model, and of the irreducible error (brown line) due to unknown elements that are not represented within the dataset. (C) When the sample size of a study is small, then the variance is large, yielding wide confidence intervals around the estimated parameter that are more likely to include the true parameter value. (D) If the same study has a larger sample size, then the reduced variance will yield more narrow confidence intervals that are less likely to include the true parameter.

Figure 3. Potential causes of the big data paradox in an RCT. (A) when the sample size of the RCT is small, the corresponding wide CIs may include the true parameter of interest, e.g. the true HR for the relative treatment effect of one treatment versus another. (B) As the sample size of the RCT increases, the quality of the gathered data may decrease, resulting in increased irreducible error within the dataset even if the enrolled patient cohort was highly homogeneous and the statistical model bias remains the same. This can be addressed by maintaining high data quality throughout the study. (C) As the sample size of the RCT increases, the heterogeneity of the patient cohort may also increase, resulting in both increased bias and increased irreducible error, even if the data quality remains consistently high throughout the study. This can be addressed by both anticipating and modeling patient heterogeneity in the estimation of the HR. (D) As the sample size of the RCT increases, the reducible error of the estimated parameter is dominated by the systematic error (bias) and is not properly represented by CIs that are a function of the standard error (variance). This occurs even if the enrolled patient cohort is highly homogeneous and data quality remains consistently high throughout the study. It can be addressed via the use of error intervals that represent the reducible error due to both systematic and standard error.