The Questionable Ecological Validity of Ecological Momentary Assessment: Considerations for Design and Analysis

REFERENCES

• Abbott, A. (1995). Sequence analysis: New methods for old ideas. Annual Review of Sociology, 21, 93–113. doi:10.1146/annurev.so.21.080195.000521
   Web of Science ®Google Scholar
• Almeida, D. M. (2005). Resilience and vulnerability to daily stressors assessed via diary methods. Current Directions in Psychological Science, 14, 64–68. doi:10.1111/j.0963-7214.2005.00336.x
   Web of Science ®Google Scholar
• Baltes, P. B., & Nesselroade, J. R. (1979). History and rationale of longitudinal research. In J. R. Nesselroade & P. B. Baltes (Eds.), Longitudinal research in the study of behavior and development (pp. 1–39). New York, NY: Academic Press.
   Google Scholar
• Barta, W. D., Tennen, H., & Litt, M. D. (2012). Measurement reactivity in diary research. In M. R. Mehl, & T. S. Conner (Eds.), Handbook of research methods for studying daily life (pp. 108–123). New York, NY: Guilford Press.
   Google Scholar
• Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579–616. doi:10.1146/annurev.psych.54.101601.145030
   PubMed Web of Science ®Google Scholar
• Bolger, N., & Laurenceau, J. P. (2013). Intensive longitudinal methods: An introduction to diary and experience sampling research. New York, NY: Guilford Press.
   Google Scholar
• Brinberg, M., Ram, N., Hülür, G., Brick, T. R., & Gerstorf, D. (in press). Analyzing dyadic data using grid-sequence analysis: Inter-dyad differences in intra-dyad dynamics. Journal of Gerontology: Psychological Sciences.
   Google Scholar
• Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 513–531. doi:10.1037/0003-066X.32.7.513
   Web of Science ®Google Scholar
• Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62, 193–217. doi:10.1037/h0047470
   PubMed Web of Science ®Google Scholar
• Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: RandMcNally.
   Google Scholar
• Corbin, J., & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Thousand Oaks, CA: Sage.
   Google Scholar
• Csikszentmihalyi, M., & Larson, R. (1984). Being adolescent: Conflict and growth in the teenage years. New York, NY: Basic Books.
   Google Scholar
• Elzinga, C. H., & Liefbroer, A. C. (2007). De-standardization of family-life trajectories of young adults: A cross-national comparison using sequence analysis. European Journal of Population, 23, 225–250. doi:10.1007/s10680-007-9133-7
   Web of Science ®Google Scholar
• Fiske, D. (1971). Measuring the concept of personality. Chicago, IL: Aldine.
   Google Scholar
• Gabadinho, A., Ritschard, G., Muller, N. S., & Studer, M. (2011). Analyzing and visualizing state sequences in R with TraMineR. Journal of Statistical Software, 40(4), 1–37. doi:10.18637/jss.v040.i04
   Web of Science ®Google Scholar
• Hamaker, E. L. (2012). Why researchers should think “within-person”: A paradigmatic rationale. In M. R. Mehl & T. S. Conner (Eds.), Handbook of research methods for studying daily life (pp. 43–61). New York, NY: Guilford Press.
   Google Scholar
• Harari, G. M., Lane, N. D., Wang, R., Crosier, B. S., Campbell, A. T., & Gosling, S. D. (2016). Using smartphones to collect behavioral data in psychological Science: Opportunities, practical considerations, and challenges. Perspectives on Psychological Science, 11, 838–854. doi:10.1177/1745691616650285
   Web of Science ®Google Scholar
• Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Epistemological foundations for the measurement of experience. In J. M. Hektner, J. A. Schmidt, & M. Csikszentmihalyi (Eds.), Experience sampling method: Measuring the quality of everyday life (pp. 3–15). Thousand Oaks, CA: Sage.
   Google Scholar
• Hormuth, S. E. (1986). The sampling of experiences in situ. Journal of Personality, 54, 262–293. doi:10.1111/jopy.1986.54.issue-1
   Web of Science ®Google Scholar
• Hovsepian, K., Al’absi, M., Ertin, E., Kamarck, T., Nakajima, M., & Kumar, S. (2015, September). cStress: Towards a gold standard for continuous stress assessment in the mobile environment. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing (pp. 493–504). New York, NY: ACM.
   Google Scholar
• Hufford, M. R., & Shields, A. L. (2002). Electronic diaries: An examination of applications and what works in the field. Applied Clinical Trials, 11, 46–56.
   Google Scholar
• Hufford, M. R. (2007). Special methodological challenges and opportunities in ecological momentary assessment. In Stone, S. Shiffman, A. A. Atienza, & L. Nebeling (Eds.), The science of real-time data capture: Self-reports in health research (pp. 54–75). Oxford University Press.
   Google Scholar
• Kearns, K., Edwards, R., & Tingstrom, D. H. (1990). Accuracy of long momentary time-sampling intervals: Implications for classroom data collection. Journal of Psychoeducational Assessment, 8, 74–85. doi:10.1177/073428299000800109
   Web of Science ®Google Scholar
• Kim, B.-R., & Teti, D. M. (2014). Maternal emotional availability during infant bedtime: An ecological framework. Journal of Family Psychology, 28, 1–11. doi:10.1037/a0035157
   PubMed Web of Science ®Google Scholar
• Koffer, R. E., Ram, N., Conroy, D. E., Pincus, A. L., & Almeida, D. (2016). Stressor diversity: Introduction and empirical integration into the daily stress model. Psychology and Aging, 31, 301–320. doi:10.1037/pag0000095
   PubMed Web of Science ®Google Scholar
• Lu, H., Frauendorfer, D., Rabbi, M., Mast, M. S., Chittaranjan, G. T., Campbell, A. T., … Choudhury, T. (2012). Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (pp. 351–360). New York, NY: ACM.
   Google Scholar
• Mark, G., Gonzalez, V. M., & Harris, J. (2005, April). No task left behind?: Examining the nature of fragmented work. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 321–330). Portland, OR: ACM.
   Google Scholar
• McLachlan, G. J., & Peel, D. (2000). Finite mixture models. New York, NY: Wiley.
   Google Scholar
• Meany‐Daboul, M. G., Roscoe, E. M., Bourret, J. C., & Ahearn, W. H. (2007). A comparison of momentary time sampling and partial‐interval recording for evaluating functional relations. Journal of Applied Behavior Analysis, 40, 501–514. doi:10.1901/jaba.2007.40-501
   PubMed Web of Science ®Google Scholar
• Mehl, M. R. (2012). Naturalistic observation sampling: The electronically activated recorder (EAR). In M. R. Mehl & T. S. Conner (Eds.), Handbook of research methods for studying daily life (pp. 176–192). New York, NY: Guilford Press.
   Google Scholar
• Mehl, M. R., & Conner, T. S. (Eds.). (2012). Handbook of research methods for studying daily life. New York, NY: Guilford Press.
   Google Scholar
• Molenaar, P. C. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement, 2, 201–218.
   Google Scholar
• Needleman, S. B., & Wunsch, C. D. (1970). A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48, 443–453. doi:10.1016/0022-2836(70)90057-4
   PubMed Web of Science ®Google Scholar
• Nesselroade, J. R., & Molenaar, P. C. (2016). Some behavioral science measurement concerns and proposals. Multivariate Behavioral Research, 51, 396–412. doi:10.1080/00273171.2015.1050481
   PubMed Web of Science ®Google Scholar
• Newcomb, M. E., Swann, G., Estabrook, R., Corden, M., Begale, M., Ashbeck, A., … Mustanski, B. (in press). Patterns and predictors of compliance in a prospective diary study of substance use and sexual behavior in a sample of young men who have sex with men. Assessment.
   Google Scholar
• Nyquist, H. (2002). Certain topics in telegraphic transmission theory. Proceedings of the IEEE, 86, 280–305. ( Reprinted from Transactions of the AIEE, pp. 617–644, 1928)
   Google Scholar
• Powell, J., Martindale, A., & Kulp, S. (1975). An evaluation of time-sample measures of behavior. Journal of Applied Behavior Analysis, 8, 463–469. doi:10.1901/jaba.1975.8-463
   PubMed Web of Science ®Google Scholar
• Powell, J., Martindale, B., Kulp, S., Martindale, A., & Bauman, R. (1977). Taking a closer look: Time sampling and measurement error. Journal of Applied Behavior Analysis, 10, 325–332. doi:10.1901/jaba.1977.10-325
   PubMed Web of Science ®Google Scholar
• Radley, K. C., O’Handley, R. D., & Labrot, Z. C. (2015). A comparison of momentary time sampling and partial-interval recording for assessment of effects of social skills training. Psychology in the Schools, 52, 363–378. doi:10.1002/pits.21829
   Web of Science ®Google Scholar
• Ram, N., Conroy, D., Pincus, A. L., Lorek, A., Rebar, A. H., Roche, M. J., & Gerstorf, D. (2014). Examining the interplay of processes across multiple time-scales: Illustration with the Intraindividual Study of Affect, Health, and Interpersonal Behavior (iSAHIB). Research in Human Development, 11, 142–160. doi:10.1080/15427609.2014.906739
   PubMed Web of Science ®Google Scholar
• Ram, N., & Gerstorf, D. (2009). Time structured and net intraindividual variability: Tools for examining the development of dynamic characteristics and processes. Psychology and Aging, 24, 778–791. doi:10.1037/a0017915
   PubMed Web of Science ®Google Scholar
• Ram, N., & Reeves, B. (in press). Time sampling. In M. Bornstein (Ed.), The SAGE encyclopedia of lifespan human development. New York, NY: Sage.
   Google Scholar
• Reis, H. T. (2012). Why researchers should think “real-world”: A conceptual rationale. In M. R. Mehl & T. S. Conner (Eds.), Handbook of research methods for studying daily life (pp. 3–21). New York, NY: Guilford Press.
   Google Scholar
• Reynolds, B. M., Robles, T. F., & Repetti, R. L. (2016). Measurement reactivity and fatigue effects in daily diary research with families. Developmental Psychology, 52, 442–456. doi:10.1037/dev0000081
   PubMed Web of Science ®Google Scholar
• Saeb, S., Zhang, M., Karr, C. J., Schueller, S. M., Corden, M. E., Kording, K. P., & Mohr, D. C. (2015). Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: An exploratory study. Journal of Medical Internet Research, 17, e175. doi:10.2196/jmir.4273
   PubMed Web of Science ®Google Scholar
• Salathé, M., Kazandjieva, M., Lee, J. W., Levis, P., Feldman, M. W., & Jones, J. H. (2010). A high-resolution human contact network for infectious disease transmission. Proceedings of the National Academy of Sciences, 107, 22020–22025.
   PubMed Web of Science ®Google Scholar
• Sankoff, D., & Kruskal, J. B. (Eds.). (1983). Time warps, string edits, and macromolecules: The theory and practice of sequence comparison. Reading, MA: Addison Wesley.
   Google Scholar
• Schwartz, N. (2007). Retrospective and concurrent self-reports: The rationale for real-time data capture. In A. Stone, S. Shiffman, A. A. Atienza, & L. Nebeling (Eds.), The science of real-time data capture: Self-reports in health research (pp. 11–26). New York, NY: Oxford University Press.
   Google Scholar
• Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. doi:10.1146/annurev.clinpsy.3.022806.091415
   PubMed Web of Science ®Google Scholar
• Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2002). Patient non-compliance with paper diaries. British Medical Journal, 324(7347), 1193–1194. doi:10.1136/bmj.324.7347.1193
   PubMed Web of Science ®Google Scholar
• Stone, A. A., Shiffman, S., Atienza, A. A., & Nebeling, L. (2007). Historical roots and rationale of ecological momentary assessment (EMA). In A. A. Stone, S. Shiffman, A. A. Atienza, & L. Nebeling (Eds.), The science of real-time data capture: Self-reports in health research (pp. 3–10). Oxford: Oxford University Press.
   Google Scholar
• Stone, A. A., & Shiffman, S. S. (2011). Ecological validity for patient reported outcomes. In A. Steptoe, K. Freedland, J. R. Jennings, M. M. Llabre, S. B. Manuck, & E. J. Susman (Eds.), Handbook of behavioral medicine: Methods and applications (pp. 99–112). New York, NY: Springer.
   Google Scholar
• Studer, M., & Ritschard, G. (2015). What matters in differences between life trajectories: A comparative review of sequence dissimilarity measures. Journal of the Royal Statistical Society, Series A, 179, 481–511. doi:10.1111/rssa.12125.
   Web of Science ®Google Scholar
• Wang, R., Chen, F., Chen, Z., Li, T., Harari, G., Tignor, S., … Campbell, A. T. (2014). StudentLife: Assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 3–14). New York, NY: ACM.
   Google Scholar
• Ward, J. H., Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236–244. doi:10.1080/01621459.1963.10500845
   Web of Science ®Google Scholar
• Wegener, D. T., & Blankenship, K. L. (2007). Ecological validity. In R. F. Baumeister & K. D. Vohs (Eds.), Encyclopedia of social psychology (pp. 275–276). Thousand Oaks, CA: Sage.
   Google Scholar
• Wilhelm, P., Perrez, M., & Pawlik, K. (2012). Conducting research in daily life: A historical review. In M. R. Mehl & T. S. Conner (Eds.), Handbook of research methods for studying daily life (pp. 62–86). New York, NY: Guilford Press.
   Google Scholar
• Yeykelis, L., Cummings, J. J., & Reeves, B. (2014). Multitasking on a single device: Arousal and the frequency, anticipation, and prediction of switching between media content on a computer. Journal of Communication, 64, 167–192. doi:10.1111/jcom.12070
   PubMed Web of Science ®Google Scholar