Discussion of ‘A selective review of statistical methods using calibration information from similar studies’

Combining information from similar studies has attracted substantial attention and continues to become increasingly important to assemble quality evidence in comparative effectiveness research. To my knowledge, this is the first paper to systematically review classical and up-to-date methods on how different statistical methods, such as meta-analysis, empirical likelihood (EL), renewal estimation and incremental inference, can be applied to incorporate information from multiple sources. This review paper succinctly presents both basic and advanced issues and will be greatly beneficial for researchers who are interested in this field.

Because of the wide array of related methods, this paper consists of cohesive but relatively independent sections. Although it is a review paper, the focus and contents are quite different from those of original papers. For example, an optimal combination of two estimators from two independent studies is derived by two methods from different perspectives: a linear combination with the smallest asymptotic variance and the maximum likelihood method. Another example is how to select a more efficient way to synthesize auxiliary information from other studies. In Section 5 of the review paper, two different sets of constraints, in which one involves parameter of interest and the other does not, have been presented and compared in terms of efficiency improvement. Both statistical intuition and theoretical justification are provided, which help readers create a better way to combine aggregate information for improved efficiency in practice. Such insightful discussions are not easily found elsewhere. The paper also nicely derives the conclusion that, similar to parametric-likelihood-based meta-analysis, the calibration methods (e.g., EL and generalized method of moments (GMM)) based on aggregate information have no efficiency loss compared to these methods using all individual data. Such deep insight into these methods greatly promotes their use for information calibration, since it is always challenging to obtain individual-level data.

As stated in the title, this review paper mainly focuses on statistical methods using calibration information from similar studies. One crucial assumption of these methods is homogeneity between the cohort with individual data (e.g., target cohort) and these similar studies (e.g., external sources). When the calibration information from the external sources are not comparable with those of the target cohort, such calibration methods may result in severe bias in estimation and misleading conclusions (Chen et al., 2021; Huang et al., 2016). One way to address this issue is to test the comparability by comparing calibration information between the target cohort and external sources before combining such information. Using the setup in Section 4 of the review paper as an example, assume that the auxiliary information from external sources is the mean of Y by subgroups (e.g., subgroups determined by covariates such as age and gender), (1) $E(Y|\mathbf{\text{X}}\in {\mathrm{\Omega }}_k)={\varphi }_k,k=1,\dots ,K.$(1) Then the compatibility can be checked by testing the difference between the sample average of outcomes within ${\mathrm{\Omega }}_k$ from the individual-level data of the target cohort and the auxiliary information from external sources using a Wald-type test. However, the conclusion of this strategy is borrowing/no borrowing, which does not fully make use of comparable/discrepant degree between the two different sources.

Chen et al. (2021) proposed a penalized constraint EL to accommodate the potential discrepancy between the target cohort and auxiliary information from external sources, and Li et al. (2022) used a similar principal in transfer learning. Basically, the penalized likelihood allows us to determine the degree of information borrowing from the external sources by making use of a penalty term with a tuning parameter. Although the method with the penalty on discrepancy was developed under the EL framework, it can be easily extended to other estimating methods such as GMM. Using the setup in Section 4 of the review paper as an illustrating example, define the mean of Y in the subgroup ${\mathrm{\Omega }}_k$ of the target cohort as ${\varphi }_k^{\ast }$, i.e., (2) $E(Y|\mathbf{\text{X}}\in {\mathrm{\Omega }}_k)={\varphi }_k^{\ast },k=1,\dots ,K.$(2) When integrating the auxiliary information with the target cohort, the comparability assumption would be violated if there existed at least one k such that ${\varphi }_k^{\ast }\ne {\varphi }_k$. We introduce a K-dimensional vector parameter $\tau \tau =({\tau }_1,\dots ,{\tau }_K{)}^{\mathrm{T}}$ to incorporate the potential incomparable information for ${\varphi }_k^{\ast }$: ${\tau }_k={\varphi }_k^{\ast }-{\varphi }_k$. We need to estimate $\tau \tau$ and identify non-zero ${\tau }_k$s to adaptively combine the auxiliary information from the external sources.

Note that Equation (2(2) $E(Y|\mathbf{\text{X}}\in {\mathrm{\Omega }}_k)={\varphi }_k^{\ast },k=1,\dots ,K.$(2) ) can be expressed equivalently to $\begin{array}{rl}& E[I(\mathbf{\text{X}}\in {\mathrm{\Omega }}_k)\{\int yf(y|\mathbf{\text{X}},\text{β}\text{β})\mathrm{d}y-{\varphi }_k^{\ast }\}|\mathbf{\text{X}}\in {\mathrm{\Omega }}_k]\\ & =0,k=1,\dots ,K.\end{array}$ Define $\mathrm{\Phi }{\mathrm{\Phi }}_n(\text{β}\text{β},\tau \tau )=\left\{g_n\right(\text{β}\text{β}{)}^{\mathrm{T}},h_n^{\left(1\right)}(\text{β}\text{β},\tau \tau ),\dots ,h_n^{\left(K\right)}(\text{β}\text{β},\tau \tau ){\}}^{\mathrm{T}}$, where $g_n\left(\text{β}\text{β}\right)=\sum _{i=1}^n{\mathrm{\nabla }}_{\text{β}}\log \left\{f\right(y_i|{\mathbf{x}}_i,\text{β}\text{β}\left)\right\}$ is the score equation, and $h_n^{\left(k\right)}(\text{β}\text{β},\tau \tau )=\sum _{i=1}^n\left[I\right({\mathbf{x}}_i\in {\mathrm{\Omega }}_k\left)\right\{\int y\times f(y|{\mathbf{x}}_{\mathbf{i}},\text{β}\text{β})\mathrm{d}y-({\varphi }_k+{\tau }_k)\left\}\right],k=1,\dots ,K,$ is the estimating equation to link the target cohort with auxiliary information. We can then construct a penalized GMM objective function, which combines the above estimating equations and a penalty term with a tuning parameter γ on the potential differences characterized by $\tau \tau$, (3) $\begin{array}{rl}{O}_n(\text{β}\text{β},\tau \tau )& =\mathrm{\Phi }{\mathrm{\Phi }}_n(\text{β}\text{β},\tau \tau {)}^{\mathrm{T}}{\mathrm{\Sigma }}_n(\text{β}{\text{β}}^{\ast },\tau {\tau }^{\ast }{)}^{-1}\mathrm{\Phi }{\mathrm{\Phi }}_n(\text{β}\text{β},\tau \tau )\\ & +\sum _{k=1}^K{p}_{\gamma }\left(|{\tau }_k|\right).\end{array}$(3) Here, ${\mathrm{\Sigma }}_n(\text{β}{\text{β}}^{\ast },\tau {\tau }^{\ast })$ is the sample variance of the estimating equations with initial but consistent estimators $(\text{β}{\text{β}}^{\ast },\tau {\tau }^{\ast })$: ${\mathrm{\Sigma }}_n(\text{β}{\text{β}}^{\ast },\tau {\tau }^{\ast })=\sum _{i=1}^n\mathrm{\Phi }{\mathrm{\Phi }}_n(\text{β}{\text{β}}^{\ast },\tau {\tau }^{\ast })\mathrm{\Phi }{\mathrm{\Phi }}_n(\text{β}{\text{β}}^{\ast },\tau {\tau }^{\ast }{)}^{\mathrm{T}}/n$. The penalty function could be the smoothly clipped absolute deviation (SCAD) penalty or least absolute shrinkage and selection operator (LASSO), which have been well studied in the literature and have many desirable properties (Fan & Li, 2001; Tibshirani, 1996). The tuning parameter $\gamma \gamma$ can be selected by minimizing the Bayesian information criterion (BIC)-like criterion (Kai et al., 2011; Wang et al., 2007). It will be interesting to investigate how this framework can be used in various methods reviewed by Qin et al. to combine information from multiple sources without assuming comparability.

Although there is a substantial amount of literature on statistical methods using calibration information from other sources, user-friendly computational tools for implementation are limited. We can look forward in the future to having well-documented and publicly available software such as R packages for practical applications in information integration, which are expected to have direct and high impact on comparative effectiveness research.

Disclosure statement

No potential conflict of interest was reported by the author(s).