Estimating the Reliability of Aggregated and Within-Person Centered Scores in Ecological Momentary Assessment

Abstract

A procedure for estimating the reliability of test scores in the context of ecological momentary assessment (EMA) was proposed to take into account the characteristics of EMA measures. Two commonly used test scores in EMA were considered: the aggregated score (AGGS) and the within-person centered score (WPCS). Conceptually, AGGS and WPCS represent the interindividual differences and the intraindividual differences, respectively. The reliability coefficients for AGGS and WPCS were derived using a multilevel factor model with a serial correlation structure framework. Point estimates and confidence intervals of these coefficients were obtained using Mx (Neale, Boker, Xie, & Maes, 2004). A simulation study showed that the proposed procedure performed well empirically. Diary data from Huang (2009), which recorded daily joy level of 110 undergraduate students for 8 days, was used to illustrate the applicability of the proposed method.

Notes

¹To construct the CIs for the reliability coefficients, several studies used the delta method (e.g., Laenen et al., 2009; Raykov & Shrout, 2002). The likelihood-based approach is another method to construct CIs in maximum likelihood estimation. The relative merits and drawbacks of the two approaches were discussed by Cheung (2009) and Neale and Miller (1997).

²If K = 2, further constraints on parameters are needed. For example, the factor loadings at each level could be constrained to be equal.