Estimating Reliability of Within-Person Couplings in a Multilevel Framework

Abstract

Within-person couplings play a prominent role in psychological research and previous studies have shown that interindividual differences in within-person couplings predict future behavior. For example, stress reactivity—operationalized as the within-person coupling of stress and positive or negative affect—is an important predictor of various (mental) health outcomes and has often been assumed to be a more or less stable personality trait. However, issues of reliability of these couplings have been largely neglected so far. In this work, we present an estimate for the reliability of within-person couplings that can be easily obtained using the user-modifiable R code accompanying this work. Results of a simulation study show that this index performs well even in the context of unbalanced data due to missing values. We demonstrate the application of this index in a measurement burst study targeting the reliability and test–retest correlation of stress reactivity estimates operationalized as within-person couplings in a daily diary design. Reliability and test–retest correlations of stress reactivity estimates were rather low, challenging the implicit assumption of stress reactivity as a stable person-level variable. We highlight key factors that researchers planning studies targeting interindividual differences in within-person couplings should consider to maximize reliability.

Acknowledgments

Part of this research was presented at the 50th meeting of the German Psychological Society in Leipzig, Germany, and the 13th Meeting of the Section for Methods and Evaluation of the German Psychological Society in Tübingen, Germany.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://osf.io/bdw5r/. To obtain the author’s disclosure form, please contact the Editor.

Notes

1 To obtain a valid estimator for the within-person association of stress and negative affect, the (continuous) time-varying predictor should be centered on the person mean (Wang & Maxwell, 2015). Unless otherwise noted, time-varying predictors have all been centered on the person mean in this work.

2 Note that the model can easily be expanded to include more than one predictor. However, when entering several predictors simultaneously, the meaning of the within-person coupling changes, and these couplings have to be interpreted as partial regression coefficients (see Discussion section for further details). An alternative approach to examine the within-person couplings of Y with several predictors is to run separate models including only one predictor at a time and examine the couplings from these models. This approach results in bivariate within-person couplings (i.e., zero-order within-person associations not controlling for other variables). Whether a simultaneous (including all predictors at once) or sequential approach (including only one predictor at a time) is more appropriate depends on the research question. In both approaches, reliability of within-person couplings can be estimated with the procedure introduced in this work.

3 Note that the original formulation by Raudenbush and Bryk (2002) looks different from Equation 4; we have adjusted the equation to better fit the notation used in this article. Further details can be found in the supplemental Appendix A.

4 In the context of structural equation modeling (SEM) latent growth curve modeling, Rast and Hofer (2014) proposed a similar measure for estimating the reliability of interindividual differences in rate of change (growth rate reliability). As we demonstrate in Appendix A (supplemental materials), this measure is a special case of the measure proposed in this work.

5 Specifically, we created a distribution with the following expected proportions of missing values: 0% missing values were expected for 18% of the participants, 10% missing values for 30% of the participants, 20% missing values for 20% of the participants, 30% missing values for 11% of the participants, 40% missing values for 9% of the participants, 50% missing values for 7% of the participants, and 60% missing values for 5% of the participants.

6 We also analyzed empirical reliabilities of the OLS estimates, ${\widehat{\mathrm{\upsilon }}}_{1i}$. Across all conditions, the reliabilities of OLS and EB estimates were nearly perfectly correlated, r = .96, and, importantly, empirical reliability was generally higher for EB estimates compared to OLS estimates; see Appendix B in the supplemental materials for detailed results regarding the OLS estimates.

7 For ease of computation, the Excel sheet in the supplemental materials can be used to arrive at this estimate.

8 The Excel sheet (tab “Estimating T”) in the supplemental materials can be used to compute the required number of measurement occasions.