Do therapist effects really impact estimates of within-patient mechanisms of change? A Monte Carlo simulation study

Figures & data

Table I. Simulation results for Person-mean centered model with balanced assignment of patients to therapists at ICC = 0, .05, .20, and .40^a.

		2-level estimates						3-level estimates
β = .30		β			SE			β			SE
NL2	NL3	Est.^b	Bias^c	Power^d	SD^e	Est.^f	Cover^g	Est.^b	Bias^c	Power^d	SD^e	Est.^f	Cover^g
50	5	0.298	−0.7%	95.6%	0.081	0.076	93.3%	0.298	−0.7%	96.6%	0.081	0.066	83.0%
50	10	0.298	−0.6%	96.2%	0.078	0.076	93.7%	0.298	−0.6%	96.2%	0.078	0.071	89.9%
100	5	0.297	−1.0%	100.0%	0.055	0.055	95.0%	0.297	−1.0%	99.9%	0.055	0.046	84.0%
100	20	0.297	−0.9%	99.9%	0.056	0.055	94.2%	0.297	−0.9%	100.0%	0.056	0.053	92.1%
200	5	0.299	−0.2%	100.0%	0.039	0.039	95.1%	0.299	−0.2%	100.0%	0.039	0.033	84.8%
200	20	0.300	−0.1%	100.0%	0.039	0.039	95.0%	0.300	−0.1%	100.0%	0.039	0.038	93.4%
200	40	0.299	−0.3%	100.0%	0.040	0.039	93.8%	0.299	−0.3%	100.0%	0.040	0.038	92.4%
β = 0		Est.^b	Bias^c	Alpha^h	SD^e	Est.^f	Cover^g	Est.^b	Bias^c	Alpha^h	SD^e	Est.^f	Cover^g
50	5	−0.001	–	6.9%	0.084	0.080	93.0%	−0.001	–	16.0%	0.084	0.068	84.0%
50	10	−0.001	–	6.4%	0.083	0.080	93.7%	−0.001	–	11.2%	0.083	0.074	88.8%
100	5	−0.003	–	5.8%	0.058	0.057	94.2%	−0.003	–	15.6%	0.058	0.049	84.4%
100	20	−0.003	–	6.0%	0.060	0.057	94.1%	−0.003	–	8.0%	0.060	0.055	92.0%
200	5	−0.001	–	4.6%	0.040	0.041	95.4%	−0.001	–	15.0%	0.040	0.035	85.0%
200	20	−0.001	–	4.5%	0.041	0.041	95.5%	−0.001	–	6.4%	0.041	0.039	93.7%
200	40	−0.001	–	6.3%	0.042	0.041	93.7%	−0.001	–	7.3%	0.042	0.040	92.7%

^aSince estimates were exactly identical for all ICC levels, only one set of results are presented in the table.

^bAverage estimate of β over 2000 replications.

^cCalculated as $(\overline{\widehat{\text{β}}}-\text{β})/\text{β}$.

^dProportion of 2000 replications yielding significant estimate when β = .30.

^eStandard deviation of β over 2000 replications (used as estimate of population standard error).

^fAverage of estimated standard errors over 2000 replications.

^gProportion of 2000 replications when 95% confidence intervals contain the population value of β.

^hProportion of 2000 replications yielding significant estimate when β = 0.

Table II. Simulation results for Person-mean centered model with unbalanced assignment of patients to therapists at ICC = 0, .05, .20, and .40.

		2-level estimates						3-level estimates
β = .30		β			SE			β			SE
NL2	NL3	Est.^b	Bias^c	Power^d	SD^e	Est.^f	Cover^g	Est.^b	Bias^c	Power^d	SD^e	Est.^f	Cover^g
50	5	0.297	−0.9%	96.1%	0.080	0.076	93.7%	0.297	−0.9%	97.5%	0.080	0.055	74.7%
50	10	0.298	−0.6%	96.7%	0.078	0.076	93.9%	0.298	−0.6%	96.9%	0.078	0.067	87.7%
100	5	0.298	−0.8%	100.0%	0.055	0.055	94.4%	0.298	−0.8%	99.9%	0.055	0.047	84.7%
100	16	0.297	−1.1%	100.0%	0.056	0.055	94.0%	0.297	−1.1%	99.9%	0.056	0.050	89.0%
200	7	0.299	−0.4%	100.0%	0.039	0.039	94.9%	0.299	−0.4%	100.0%	0.039	0.033	85.8%
200	18	0.300	−0.1%	100.0%	0.039	0.039	94.7%	0.300	−0.1%	100.0%	0.039	0.036	90.8%
β = 0		Est.^b	Bias^c	Alpha^h	SD^e	Est.^f	Cover^g	Est.^b	Bias^c	Alpha^h	SD^e	Est.^f	Cover^g
50	5	−0.003	–	6.2%	0.082	0.080	93.8%	−0.0025	–	24.0%	0.082	0.057	76.0%
50	10	−0.002	–	6.4%	0.082	0.080	93.6%	−0.0021	–	12.9%	0.082	0.070	87.1%
100	5	−0.002	–	5.4%	0.058	0.057	94.7%	−0.0018	–	14.7%	0.058	0.049	85.3%
100	16	−0.003	–	6.4%	0.059	0.057	93.6%	−0.0032	–	11.3%	0.059	0.052	88.7%
200	7	−0.001	–	6.0%	0.041	0.041	94.0%	−0.0014	–	15.8%	0.041	0.035	84.2%
200	18	0.000	–	5.4%	0.041	0.041	94.6%	−0.0001	–	8.6%	0.041	0.038	91.4%

^aSince estimates were exactly identical for all ICC levels, only one set of results are presented in the table.

^bAverage estimate of β over 2000 replications.

^cCalculated as $(\overline{\widehat{\text{β}}}-\text{β})/\text{β}$.

^dProportion of 2000 replications yielding significant estimate when β = .30.

^eStandard deviation of β over 2000 replications (used as estimate of population standard error).

^fAverage of estimated standard errors over 2000 replications.

^gProportion of 2000 replications when 95% confidence intervals contain the population value of β.

^hProportion of 2000 replications yielding significant estimate when β = 0.

Table III. Simulation results for Person-mean centered model with therapist-level random slopes (slope SD = 0.15).

		2-level estimates						3-level estimates
ICCslope = .05		β			SE			β			SE
NL2	NL3	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.299	−0.2%	94.8%	0.083	0.081	93.3%	0.300	−0.1%	91.4%	0.083	0.087	85.9%
50	10	0.298	−0.8%	95.1%	0.082	0.081	94.1%	0.298	−0.8%	89.6%	0.083	0.111	91.9%
100	5	0.299	−0.4%	99.8%	0.061	0.058	92.7%	0.299	−0.3%	98.9%	0.061	0.055	84.0%
100	20	0.299	−0.2%	100.0%	0.060	0.058	94.1%	0.299	−0.3%	98.9%	0.059	0.063	93.2%
200	5	0.299	−0.2%	100.0%	0.045	0.041	92.8%	0.299	−0.2%	99.8%	0.045	0.038	84.2%
200	20	0.300	0.1%	100.0%	0.042	0.041	94.4%	0.300	0.1%	99.8%	0.042	0.041	93.2%
200	40	0.301	0.4%	100.0%	0.042	0.041	93.9%	0.301	0.4%	99.9%	0.042	0.042	93.8%
ICCslope = .10	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.299	−0.2%	94.5%	0.084	0.081	92.9%	0.300	−0.1%	91.0%	0.084	0.119	85.9%
50	10	0.298	−0.7%	94.9%	0.083	0.081	93.9%	0.298	−0.7%	89.6%	0.083	0.164	92.0%
100	5	0.299	−0.4%	99.8%	0.063	0.058	92.4%	0.299	−0.3%	98.5%	0.063	0.058	84.8%
100	20	0.299	−0.2%	100.0%	0.060	0.058	94.2%	0.299	−0.2%	99.1%	0.060	0.061	93.1%
200	5	0.299	−0.3%	100.0%	0.048	0.041	91.0%	0.299	−0.3%	99.9%	0.047	0.039	85.2%
200	20	0.300	0.1%	100.0%	0.043	0.041	94.2%	0.300	0.1%	99.8%	0.043	0.044	93.2%
200	40	0.301	0.4%	100.0%	0.042	0.041	93.6%	0.301	0.4%	100.0%	0.043	0.042	93.8%
ICCslope = .20	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.300	−0.2%	94.2%	0.087	0.081	92.4%	0.300	−0.1%	89.2%	0.087	0.100	85.8%
50	10	0.298	−0.5%	94.7%	0.084	0.081	93.8%	0.298	−0.5%	89.3%	0.084	0.105	92.2%
100	5	0.299	−0.3%	99.7%	0.066	0.058	91.0%	0.299	−0.3%	98.5%	0.066	0.063	84.7%
100	20	0.300	−0.2%	100.0%	0.061	0.058	93.8%	0.299	−0.2%	98.7%	0.060	0.066	93.3%
200	5	0.299	−0.4%	100.0%	0.052	0.041	87.7%	0.299	−0.4%	99.8%	0.052	0.044	85.3%
200	20	0.300	0.1%	100.0%	0.044	0.041	93.5%	0.300	0.1%	99.8%	0.044	0.043	93.0%
200	40	0.301	0.4%	100.0%	0.043	0.041	93.3%	0.301	0.4%	99.9%	0.043	0.042	93.7%

^aAverage estimate of β over 2000 replications.

^bCalculated as $(\overline{\widehat{\text{β}}}-\text{β})/\text{β}$.

^cProportion of 2000 replications yielding significant estimate when β = .30.

^dStandard deviation of β over 2000 replications (used as estimate of population standard error).

^eAverage of estimated standard errors over 2000 replications.

^fCalculated as $(\overline{\widehat{se}}-sd)/sd$.

^gProportion of 2000 replications when 95% confidence intervals contain the population value of β.

Table IV. Simulation results for Random Intercept Cross-Lagged Panel Model with unbalanced assignment of patients to therapists at ICC = 0, .05, .20, and .40 and β = .30.

		2-level estimates						3-level estimates
ICC = 0		β			SE			β			SE
NL2	NL3	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.287	−4.3%	84.4%	0.097	0.093	93.3%	0.289	−3.8%	88.1%	0.095	20.887	78.0%
50	10	0.286	−4.6%	84.4%	0.095	0.093	93.8%	0.288	−4.1%	83.3%	0.094	11.554	87.8%
100	5	0.295	−1.7%	99.1%	0.064	0.065	95.1%	0.295	−1.6%	98.5%	0.064	0.057	87.0%
100	16	0.296	−1.5%	98.4%	0.066	0.065	94.3%	0.296	−1.4%	97.9%	0.066	0.061	90.2%
200	7	0.298	−0.7%	100.0%	0.044	0.045	95.6%	0.298	−0.6%	100.0%	0.044	0.039	86.4%
200	18	0.298	−0.8%	100.0%	0.044	0.045	96.1%	0.298	−0.8%	100.0%	0.044	0.043	92.7%
ICC = .05	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.287	−4.2%	84.4%	0.097	0.093	93.2%	0.289	−3.8%	87.3%	0.096	0.370	78.0%
50	10	0.287	−4.5%	84.3%	0.094	0.093	94.0%	0.288	−4.0%	81.8%	0.095	>100	88.0%
100	5	0.295	−1.6%	99.0%	0.065	0.065	95.0%	0.295	−1.5%	97.9%	0.064	0.061	87.4%
100	16	0.296	−1.5%	98.6%	0.066	0.065	94.3%	0.296	−1.5%	97.2%	0.066	0.077	90.8%
200	7	0.298	−0.7%	100.0%	0.045	0.045	95.2%	0.298	−0.6%	100.0%	0.044	0.039	86.3%
200	18	0.298	−0.8%	100.0%	0.045	0.045	96.1%	0.298	−0.8%	99.8%	0.044	0.045	92.9%
ICC = .20	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.288	−3.9%	85.2%	0.096	0.092	92.9%	0.290	−3.3%	88.5%	0.094	4.853	78.0%
50	10	0.288	−4.1%	84.9%	0.094	0.093	94.1%	0.289	−3.5%	84.1%	0.094	>100	87.7%
100	5	0.296	−1.5%	98.8%	0.065	0.065	94.8%	0.296	−1.2%	98.5%	0.063	>100	87.6%
100	16	0.296	−1.5%	98.4%	0.067	0.065	94.3%	0.296	−1.2%	97.6%	0.065	0.061	90.5%
200	7	0.298	−0.7%	100.0%	0.045	0.045	95.2%	0.299	−0.5%	99.9%	0.044	0.039	86.2%
200	18	0.298	−0.8%	100.0%	0.045	0.045	95.6%	0.298	−0.6%	100.0%	0.044	0.042	92.9%
ICC = .40	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.290	−3.3%	86.6%	0.095	0.091	93.3%	0.294	−2.2%	90.6%	0.090	20.178	77.2%
50	10	0.289	−3.6%	85.4%	0.093	0.092	94.0%	0.293	−2.2%	87.2%	0.091	>100	87.6%
100	5	0.296	−1.2%	99.1%	0.065	0.064	94.1%	0.298	−0.6%	99.0%	0.061	0.054	86.6%
100	16	0.296	−1.4%	98.6%	0.067	0.065	94.2%	0.298	−0.5%	98.9%	0.064	0.059	89.7%
200	7	0.298	−0.6%	100.0%	0.045	0.045	95.5%	0.300	−0.2%	99.9%	0.043	0.037	85.9%
200	18	0.298	−0.7%	100.0%	0.045	0.045	95.4%	0.299	−0.3%	100.0%	0.043	0.041	92.6%

^aAverage estimate of β over 2000 replications.

^bCalculated as $(\overline{\widehat{\text{β}}}-\text{β})/\text{β}$.

^cProportion of 2000 replications yielding significant estimate when β = .30.

^dStandard deviation of β over 2000 replications (used as estimate of population standard error).

^eAverage of estimated standard errors over 2000 replications.

^fCalculated as $(\overline{\widehat{se}}-sd)/sd$.

^gProportion of 2000 replications when 95% confidence intervals contain the population value of β.

^hProportion of 2000 replications yielding significant estimate when β = 0.

Table V. Two-level simulation results for Random Intercept Cross-Lagged Panel Model, ignoring therapist-level random slopes.

		Slope SD = 0.10						Slope SD = 0.15
ICCslope = .05		β			SE			β			SE
NL2	NL3	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.284	−5.3%	83.1%	0.100	0.093	92.7%	0.283	−5.8%	81.2%	0.103	0.094	92.2%
50	10	0.286	−4.8%	82.8%	0.102	0.093	92.4%	0.285	−5.1%	82.2%	0.105	0.094	91.6%
100	5	0.296	−1.3%	98.7%	0.068	0.065	93.8%	0.296	−1.5%	98.3%	0.071	0.066	92.5%
100	20	0.295	−1.6%	98.5%	0.068	0.065	94.0%	0.294	−1.9%	98.1%	0.070	0.066	93.8%
200	5	0.297	−0.9%	100.0%	0.048	0.046	94.2%	0.296	−1.2%	100.0%	0.050	0.046	92.6%
200	20	0.297	−1.0%	100.0%	0.047	0.046	94.7%	0.296	−1.3%	100.0%	0.049	0.046	93.8%
200	40	0.297	−1.0%	100.0%	0.047	0.046	94.7%	0.296	−1.3%	100.0%	0.049	0.046	93.7%
ICCslope = .10	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.284	−5.3%	82.7%	0.100	0.093	92.4%	0.283	−5.8%	80.8%	0.105	0.094	91.9%
50	10	0.286	−4.8%	82.7%	0.102	0.093	92.2%	0.284	−5.2%	82.5%	0.105	0.094	91.5%
100	5	0.296	−1.3%	98.7%	0.069	0.065	93.3%	0.296	−1.4%	98.1%	0.073	0.066	91.9%
100	20	0.295	−1.6%	98.5%	0.068	0.065	94.2%	0.294	−1.9%	98.2%	0.070	0.066	93.4%
200	5	0.297	−0.9%	100.0%	0.049	0.046	93.3%	0.296	−1.2%	100.0%	0.053	0.046	91.4%
200	20	0.297	−1.0%	100.0%	0.047	0.046	94.4%	0.296	−1.3%	100.0%	0.049	0.046	93.8%
200	40	0.297	−1.0%	100.0%	0.047	0.046	94.3%	0.296	−1.3%	100.0%	0.049	0.046	93.8%
ICCslope = .20	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g	Est.^a	Bias^b	Power^c	SD^d	Est.^e	Cover^g
50	5	0.284	−5.4%	82.2%	0.102	0.093	92.0%	0.283	−5.7%	80.5%	0.107	0.094	91.4%
50	10	0.285	−4.9%	83.0%	0.102	0.093	92.2%	0.284	−5.3%	81.7%	0.106	0.094	91.1%
100	5	0.296	−1.2%	98.3%	0.070	0.065	92.7%	0.296	−1.3%	97.8%	0.0754	0.0658	91.2%
100	20	0.295	−1.7%	98.5%	0.068	0.065	94.2%	0.294	−1.9%	98.0%	0.0705	0.066	93.3%
200	5	0.297	−0.9%	100.0%	0.051	0.046	91.8%	0.296	−1.2%	100.0%	0.0567	0.0461	89.3%
200	20	0.297	−1.0%	100.0%	0.047	0.046	94.6%	0.296	−1.2%	100.0%	0.0502	0.0463	93.4%
200	40	0.297	−1.0%	100.0%	0.047	0.046	94.3%	0.296	−1.3%	100.0%	0.0493	0.0462	93.3%

^aAverage estimate of β over 2000 replications.

^bCalculated as $(\overline{\widehat{\text{β}}}-\text{β})/\text{β}$.

^cProportion of 2000 replications yielding significant estimate when β = .30.

^dStandard deviation of β over 2000 replications (used as estimate of population standard error).

^eAverage of estimated standard errors over 2000 replications.

^fCalculated as $(\overline{\widehat{se}}-sd)/sd$.

^gProportion of 2000 replications when 95% confidence intervals contain the population value of β.

^hProportion of 2000 replications yielding significant estimate when β = 0.