Broadening Convenience Samples to Advance Theoretical Progress and Avoid Bias in Developmental Science

References

• Akhtar, N., & Jaswal, V. K. (2013). Deficit or difference? Interpreting diverse developmental paths: An introduction to the special section. Developmental Psychology, 49(1), 1–3. doi:10.1037/a0029851
   PubMed Web of Science ®Google Scholar
• Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C., & Cristia, A. (2018). Promoting replicability in developmental research through meta-analyses: Insights from language acquisition research. Child Development, 89(6), 1996–2009. doi:10.1111/cdev.13079
   PubMed Web of Science ®Google Scholar
• Bornstein, M. H., Jager, J., & Putnick, D. L. (2013). Sampling in developmental science: Situations, shortcomings, solutions, and standards. Developmental Review, 33(4), 357–370. doi:10.1016/j.dr.2013.08.003
   PubMed Web of Science ®Google Scholar
• Bosten, J. M. (2022). Do you see what I see? Diversity in human color perception. Annual Review of Vision Science, 8(1), 101–133. doi:10.1146/annurev-vision-093020-112820
   PubMedGoogle Scholar
• Bryan, C. J., Tipton, E., & Yeager, D. S. (2021). Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature Human Behaviour, 5(8), 980–989. doi:10.1038/s41562-021-01143-3
   PubMed Web of Science ®Google Scholar
• Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13(2), 149–154. doi:10.1177/1745691617706516
   Web of Science ®Google Scholar
• Button, K. S., Ioannidis, J., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. doi:10.1038/nrn3475
   PubMed Web of Science ®Google Scholar
• Byers-Heinlein, K., Tsui, A. S. M., Bergmann, C., Black, A. K., Brown, A., Carbajal, M. J. … Wermelinger, S. (2021). A multilab study of bilingual infants: Exploring the preference for infant-directed speech. Advances in Methods and Practices in Psychological Science, 4(1), 2515245920974622. doi:10.1177/2515245920974622
   Google Scholar
• Chessa, D., Lis, A., Riso, D. D., Delvecchio, E., Mazzeschi, C., Russ, S. W., & Dillon, J. (2013). A cross-cultural comparison of pretend play in U.S. and Italian children. Journal of Cross-Cultural Psychology, 44(4), 640–656. doi:10.1177/0022022112461853
   Web of Science ®Google Scholar
• Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464–473. doi:10.1177/1948550619875149
   Web of Science ®Google Scholar
• Chuey, A., Asaba, M., Bridgers, S., Carrillo, B., Dietz, G., Garcia, T. & Gweon, H. (2021). Moderated online data-collection for developmental research: Methods and replications. Frontiers in Psychology, 4968. doi:10.3389/fpsyg.2021.734398
   Google Scholar
• Chuey, A., Boyce, V., Cao, A., & Frank, M. C. Conducting developmental research online vs. in-person: A meta-analysis. Manuscript under review. 10.31234/osf.io/qc6fw
   Google Scholar
• Coles, N. A., Hamlin, J. K., Sullivan, L. L., Parker, T. H., & Altschul, D. (2022). Build up big-team science. Nature, 601(7894), 505–507. doi:10.1038/d41586-022-00150-2
   PubMed Web of Science ®Google Scholar
• Coppock, A., Leeper, T. J., & Mullinix, K. J. (2018). Generalizability of heterogeneous treatment effect estimates across samples. Proceedings of the National Academy of Sciences, 115(49), 12441–12446. doi:10.1073/pnas.1808083115
   PubMed Web of Science ®Google Scholar
• DeJesus, J. M., Callanan, M. A., Solis, G., & Gelman, S. A. (2019). Generic language in scientific communication. Proceedings of the National Academy of Sciences, 116(37), 18370–18377. doi:10.1073/pnas.1817706116
   PubMed Web of Science ®Google Scholar
• deMayo, B., Kellier, D., Braginsky, M., Bergmann, C., Hendriks, C., Rowland, C. F. … Marchman, V. (2021). Web-CDI: A system for online administration of the MacArthur-bates communicative development inventories. Language Development Research.
   Google Scholar
• Doebel, S. (2020). Rethinking executive function and its development. Perspectives on Psychological Science, 15(4), 942–956. doi:10.1177/1745691620904771
   Web of Science ®Google Scholar
• Doebel, S., & Lillard, A. S. (2023). How does play foster development? A new executive function perspective. Developmental Review, 67, 101064. doi:10.1016/j.dr.2022.101064
   Web of Science ®Google Scholar
• Doebel, S., & Munakata, Y. (2018). Group influences on engaging self-control: Children delay gratification and value it more when their in-group delays and their out-group doesn’t. Psychological Science, 29(5), 738–748. doi:10.1177/0956797617747367
   PubMed Web of Science ®Google Scholar
• Farver, J. A. M., & Shin, Y. L. (1997). Social pretend play in Korean- and Anglo-American preschoolers. Child Development, 68(3), 544–556. doi:10.2307/1131677
   PubMed Web of Science ®Google Scholar
• Fong, F. T., Imuta, K., Redshaw, J., & Nielsen, M. (2021). The digital social partner: Preschool children display stronger imitative tendency in screen‐based than live learning. Human Behavior and Emerging Technologies, 3(4), 585–594. doi:10.1002/hbe2.280
   Google Scholar
• Frank, M. C. (2019). N-best evaluation for academic hiring and promotion. Trends in Cognitive Sciences, 23(12), 983–985. doi:10.1016/j.tics.2019.09.010
   PubMed Web of Science ®Google Scholar
• Frank, M. C., Bergelson, E., Bergmann, C., Cristia, A., Floccia, C., Gervain, J. & Yurovsky, D. (2017). A collaborative approach to infant research: Promoting reproducibility, best practices, and theory‐building. Infancy, 22(4), 421–435. doi:10.1111/infa.12182
   PubMed Web of Science ®Google Scholar
• Frank, M. C., Braginsky, M., Yurovsky, D., & Marchman, V. A. (2021). Variability and consistency in early language learning: The wordbank project. Cambridge, MA: MIT Press.
   Google Scholar
• Gaskins, S. (2013). Pretend play as culturally constructed activity. The Oxford Handbook of the Development of Imagination, 224–247.
   Google Scholar
• Gaskins, S., & Alcalá, L. (2023). Studying executive function in culturally meaningful ways. Journal of Cognition and Development, 24(2), 260–279. doi:10.1080/15248372.2022.2160722
   Web of Science ®Google Scholar
• Gilmore, R. O., & Adolph, K. E. (2017). Video can make behavioural science more reproducible. Nature Human Behaviour, 1(7), 1–2. doi:10.1038/s41562-017-0128
   Google Scholar
• Haight, W. L., Wang, X. L., Fung, H. H. T., Williams, K., & Mintz, J. (1999). Universal, developmental, and variable aspects of young children’s play: A cross‐cultural comparison of pretending at home. Child Development, 70(6), 1477–1488. doi:10.1111/1467-8624.00107
   PubMed Web of Science ®Google Scholar
• Hamlin, J. K., Wynn, K., & Bloom, P. (2007). Social evaluation by preverbal infants. Nature, 450(7169), 557–559. doi:10.1038/nature06288
   PubMed Web of Science ®Google Scholar
• Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. doi:10.1017/S0140525X0999152X
   PubMed Web of Science ®Google Scholar
• Keller, H. (2018). Universality claim of attachment theory: Children’s socioemotional development across cultures. Proceedings of the National Academy of Sciences, 115(45), 11414–11419. doi:10.1073/pnas.1720325115
   PubMed Web of Science ®Google Scholar
• Kidd, E., & Garcia, R. (2022). How diverse is child language acquisition research? First Language, 42(6), 01427237211066405.
   Web of Science ®Google Scholar
• Kidd, C., Palmeri, H., & Aslin, R. N. (2013). Rational snacking: Young children’s decision-making on the marshmallow task is moderated by beliefs about environmental reliability. Cognition, 126(1), 109–114. doi:10.1016/j.cognition.2012.08.004
   PubMed Web of Science ®Google Scholar
• Kirkorian, H. L. (2018). When and how do interactive digital media help children connect what they see on and off the screen? Child Development Perspectives, 12(3), 210–214. doi:10.1111/cdep.12290
   Web of Science ®Google Scholar
• Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr, Alper, S. & Sowden, W. (2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. doi:10.1177/2515245918810225
   Google Scholar
• Lamm, B., Keller, H., Teiser, J., Gudi, H., Yovsi, R. D., Freitag, C. & Lohaus, A. (2018). Waiting for the second treat: Developing culture‐specific modes of self‐regulation. Child Development, 89(3), e261–e277. doi:10.1111/cdev.12847
   PubMed Web of Science ®Google Scholar
• Lourenco, S. F., & Tasimi, A. (2020). No participant left behind: Conducting science during COVID-19. Trends in Cognitive Sciences, 24(8), 583–584. doi:10.1016/j.tics.2020.05.003
   PubMed Web of Science ®Google Scholar
• MacWhinney, B. (2014). The CHILDES project: Tools for analyzing talk. Volume II: The database. Psychology Press.
   Google Scholar
• ManyBabies Consortium. (2020). Quantifying sources of variability in infancy research using the infant-directed-speech preference. Advances in Methods and Practices in Psychological Science, 3(1), 24–52. doi:10.1177/2515245919900809
   Google Scholar
• Miller-Cotto, D., Smith, L. V., Wang, A. H., & Ribner, A. D. (2022). Changing the conversation: A culturally responsive perspective on executive functions, minoritized children and their families. Infant and Child Development, 31(1), e2286. doi:10.1002/icd.2286
   Web of Science ®Google Scholar
• Morgan, A. C., LaBerge, N., Larremore, D. B., Galesic, M., Brand, J. E., & Clauset, A. (2022). Socioeconomic roots of academic faculty. Nature Human Behaviour, 6(12), 1625–1633. doi:10.1038/s41562-022-01425-4
   PubMed Web of Science ®Google Scholar
• Munakata, Y., & Michaelson, L. E. (2021). Executive functions in social context: Implications for conceptualizing, measuring, and supporting developmental trajectories. Annual Review of Developmental Psychology, 3(1), 139–163. doi:10.1146/annurev-devpsych-121318-085005
   Google Scholar
• Munakata, Y., Yanaoka, K., Doebel, S., Guild, R. M., Michaelson, L. E. … Moses, L. (2020). Group influences on children’s delay of gratification: Testing the roles of culture and personal connections. Collabra: Psychology, 6(1). doi:10.1525/collabra.265
   Google Scholar
• Nielsen, M., Haun, D., Kärtner, J., & Legare, C. H. (2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38. doi:10.1016/j.jecp.2017.04.017
   PubMed Web of Science ®Google Scholar
• Rizzo, M. T., Britton, T. C., & Rhodes, M. (2022). Developmental origins of anti-Black bias in white children in the United States: Exposure to and beliefs about racial inequality. Proceedings of the National Academy of Sciences, 119(47), e2209129119. doi:10.1073/pnas.2209129119
   PubMed Web of Science ®Google Scholar
• Roberts, S. O., Bareket-Shavit, C., Dollins, F. A., Goldie, P. D., & Mortenson, E. (2020). Racial inequality in psychological research: Trends of the past and recommendations for the future. Perspectives on Psychological Science, 15(6), 1295–1309. doi:10.1177/1745691620927709
   Web of Science ®Google Scholar
• Rogoff, B., Coppens, A. D., Alcalá, L., Aceves-Azuara, I., Ruvalcaba, O., López, A., & Dayton, A. (2017). Noticing learners’ strengths through cultural research. Perspectives on Psychological Science, 12(5), 876–888. doi:10.1177/1745691617718355
   Web of Science ®Google Scholar
• Rosen, M. L., Hagen, M. P., Lurie, L. A., Miles, Z. E., Sheridan, M. A., Meltzoff, A. N., & McLaughlin, K. A. (2020). Cognitive stimulation as a mechanism linking socioeconomic status with executive function: A longitudinal investigation. Child Development, 91(4), e762–e779. doi:10.1111/cdev.13315
   PubMed Web of Science ®Google Scholar
• Scott, K., Chu, J., & Schulz, L. (2017). Lookit (part 2): Assessing the viability of online developmental research, results from three case studies. Open Mind, 1(1), 15–29. doi:10.1162/OPMI_a_00001
   Google Scholar
• Scott, K., & Schulz, L. (2017). Lookit (part 1): A new online platform for developmental research. Open Mind, 1(1), 4–14. doi:10.1162/OPMI_a_00002
   Google Scholar
• Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51(3), 515. doi:10.1037/0022-3514.51.3.515
   Web of Science ®Google Scholar
• Sheskin, M., Scott, K., Mills, C. M., Bergelson, E., Bonawitz, E., Spelke, E. S. & Schulz, L. (2020). Online developmental science to foster innovation, access, and impact. Trends in Cognitive Sciences, 24(9), 675–678. doi:10.1016/j.tics.2020.06.004
   PubMed Web of Science ®Google Scholar
• Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123–1128. doi:10.1177/1745691617708630
   Web of Science ®Google Scholar
• Singh, L. (2022). From information to action: A commentary on Kidd and Garcia (2022). First Language, 42(6), 814–817.
   Google Scholar
• Singh, L., Barokova, M., Baumgartner, H., Lopera, D., Omane, P. O., Sheskin, M. & Frank, M. C. (2022). A unified approach to demographic data collection for research with young children across diverse cultures. Retrieved from https://psyarxiv.com/agt3d/
   Google Scholar
• Singh, L., Rajendra, S. J., & Mazuka, R. (2022). Diversity and representation in studies of infant perceptual narrowing. Child Development Perspectives, 16(4), 191–199. doi:10.1111/cdep.12468
   Web of Science ®Google Scholar
• Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020). Got bots? Practical recommendations to protect online survey data from bot attacks. The Quantitative Methods for Psychology, 16(5), 472–481. doi:10.20982/tqmp.16.5.p472
   Web of Science ®Google Scholar
• Tardif, T., Gelman, S. A., & Xu, F. (1999). Putting the “noun bias” in context: A comparison of English and mandarin. Child Development, 70(3), 620–635. doi:10.1111/1467-8624.00045
   Web of Science ®Google Scholar
• Thibodeau-Nielsen, R. B., Gilpin, A. T., Nancarrow, A. F., Pierucci, J. M., & Brown, M. M. (2020). Fantastical pretense’s effects on executive function in a diverse sample of preschoolers. Journal of Applied Developmental Psychology, 68, 101137. doi:10.1016/j.appdev.2020.101137
   Web of Science ®Google Scholar
• Thompson, B. N., & Goldstein, T. R. (2019). Disentangling pretend play measurement: Defining the essential elements and developmental progression of pretense. Developmental Review, 52, 24–41. doi:10.1016/j.dr.2019.100867
   Web of Science ®Google Scholar
• Tipton, E. (2013). Stratified sampling using cluster analysis: A sample selection strategy for improved generalizations from experiments. Evaluation Review, 37(2), 109–139. doi:10.1177/0193841X13516324
   PubMed Web of Science ®Google Scholar
• Tipton, E., & Olsen, R. B. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47(8), 516–524. doi:10.3102/0013189X18781522
   Web of Science ®Google Scholar
• Tsui, A. S. M., Carstensen, A., Kachergis, G., Abubakar, A., Asnake, M., & Barry, O. & Frank, M. C. (2022). Exploring variation in infants’ preference for infant-directed speech: Evidence from a multi-site study in Africa. Stage 1 registered report, accepted pending data collection.
   Google Scholar
• Way, S. F., Morgan, A. C., Larremore, D. B., & Clauset, A. (2019). Productivity, prominence, and the effects of academic environment. Proceedings of the National Academy of Sciences, 116(22), 10729–10733. doi:10.1073/pnas.1817431116
   PubMed Web of Science ®Google Scholar
• Webb, M. A., & Tangney, J. P. (2022). Too good to be true: Bots and bad data from mechanical Turk. Perspectives on Psychological Science, 17456916221120027. doi:10.1177/17456916221120027
   Web of Science ®Google Scholar
• Weisberg, D. S. (2015). Pretend play. Wiley Interdisciplinary Reviews: Cognitive Science, 6(3), 249–261. doi:10.1002/wcs.1341
   PubMed Web of Science ®Google Scholar
• White, R. E., & Carlson, S. M. (2016). What would Batman do? self-distancing improves executive function in young children. Developmental Science, 19(3), 419–426. doi:10.1111/desc.12314
   PubMed Web of Science ®Google Scholar
• Yanaoka, K., Michaelson, L. E., Guild, R. M., Dostart, G., Yonehiro, J., Saito, S., & Munakata, Y. (2022). Cultures crossing: The power of habit in delaying gratification. Psychological Science, 33(7), 1172–1181. doi:10.1177/09567976221074650
   PubMed Web of Science ®Google Scholar