Publication Cover
Aging, Neuropsychology, and Cognition
A Journal on Normal and Dysfunctional Development
Volume 27, 2020 - Issue 5
2,710
Views
4
CrossRef citations to date
0
Altmetric
Original Article

Relevance of working memory for reinforcement learning in older adults varies with timescale of learning

&
Pages 654-676 | Received 09 Oct 2018, Accepted 02 Sep 2019, Published online: 22 Sep 2019

References

  • Baddeley, A. (2012). Working memory: Theories, models, and controversies. Annual Review of Psychology, 63, 1–29. doi:10.1146/annurev-psych-120710-100422.
  • Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In G. A. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (pp. 47–89). New York, NY: Academic. http://books.google.com/books?hl=nl‎&id=o5LScJ9ecGUC&pgis=1
  • Balota, D. A., Duchek, J. M., & Paullin, R. (1989). Age-related differences in the impact of spacing, lag, and retention interval. Psychology and Aging, 4(1), 3–9. Retrieved from http://dx.doi.org/10.1037/0882-7974.4.1.3
  • Balota, D. A., Duchek, J. M., Sergent-Marshall, S. D., & Roediger, H. L. (2006). Does expanded retrieval produce benefits over equal-interval spacing? Explorations of spacing effects in healthy aging and early stage Alzheimer’s disease. Psychology and Aging, 21(1), 19–31. doi:10.1037/0882-7974.21.1.19
  • Bannon, M. J., & Roth, R. H. (1983). Pharmacology of mesocortical dopamine neurons. Pharmacological Reviews, 35(1), 53–68. Retrieved from http://pharmrev.aspetjournals.org/content/35/1/53.short
  • Bennett, I. J., Madden, D. J., Vaidya, C. J., Howard, D. V., & Howard, J. H., Jr. (2010). Age-related differences in multiple measures of white matter integrity: A diffusion tensor imaging study of healthy aging. Human Brain Mapping, 31(3), 378–390. doi:10.1002/hbm.20872
  • Bor, D., Duncan, J., Wiseman, R. J., & Owen, A. M. (2003). Encoding strategies dissociate prefrontal activity from working memory demand. Neuron, 37(2), 361–367. doi:10.1016/S0896-6273(02)01171-6
  • Burke, W. J., Roccaforte, W. H., & Wengel, S. P. (1991). The short form of the geriatric depression scale: A comparison with the 30-item form. Journal of Geriatric Psychiatry and Neurology, 4(3), 173–178. doi:10.1177/089198879100400310
  • Burzynska, A. Z., Nagel, I. E., Preuschhof, C., Gluth, S., Bäckman, L., Li, S.-C., … Heekeren, H. R. (2012). Cortical thickness is linked to executive functioning in adulthood and aging. Human Brain Mapping, 33(7), 1607–1620. doi:10.1002/hbm.21311
  • Burzynska, A. Z., Preuschhof, C., Bäckman, L., Nyberg, L., Li, S.-C., Lindenberger, U., & Heekeren, H. R. (2010). Age-related differences in white matter microstructure: Region-specific patterns of diffusivity. NeuroImage, 49(3), 2104–2112. doi:10.1016/j.neuroimage.2009.09.041
  • Charlton, R. A., Barrick, T. R., Lawes, I. N. C., Markus, H. S., & Morris, R. G. (2010). White matter pathways associated with working memory in normal aging. Cortex, 46(4), 474–489. doi:10.1016/J.CORTEX.2009.07.005
  • Chowdhury, R., Guitart-Masip, M., Lambert, C., Dayan, P., Huys, Q., Düzel, E., & Dolan, R. J. (2013). Dopamine restores reward prediction errors in old age. Nature Neuroscience, 16(5), 648–653. doi:10.1038/nn.3364
  • Cohen, J. D., Perlstein, W. M., Braver, T. S., Nystrom, L. E., Noll, D. C., Jonides, J., & Smith, E. E. (1997). Temporal dynamics of brain activation during a working memory task. Nature, 386(6625), 604–608. doi:10.1038/386604a0
  • Collins, A. G. E., Brown, J. K., Gold, J. M., Waltz, J. A., & Frank, M. J. (2014). Working memory contributions to reinforcement learning impairments in schizophrenia. The Journal of Neuroscience, 34(41), 13747–13756. doi:10.1523/JNEUROSCI.0989-14.2014
  • Collins, A. G. E., Ciullo, B., Frank, M. J., & Badre, D. (2017). Working memory load strengthens reward prediction errors. The Journal of Neuroscience, 37(16), 4332–4342. doi:10.1523/JNEUROSCI.2700-16.2017
  • Collins, A. G. E., & Frank, M. J. (2012). How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. The European Journal of Neuroscience, 35(7), 1024–1035. doi:10.1111/j.1460-9568.2011.07980.x
  • Conway, A. R. A., Kane, M. J., Bunting, M. F., Hambrick, D. Z., Wilhelm, O., & Engle, R. W. (2005). Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review, 12(5), 769–786. doi:10.3758/BF03196772
  • Cools, R., & D’Esposito, M. (2011). Inverted-U-shaped dopamine actions on human working memory and cognitive control. Biological Psychiatry, 69(12), e113–e125. doi:10.1016/j.biopsych.2011.03.028
  • Cowan, N. (2010). The magical mystery four. Current Directions in Psychological Science, 19(1), 51–57. doi:10.1177/0963721409359277
  • Curtis, C. E., & D’Esposito, M. (2003). Persistent activity in the prefrontal cortex during working memory. Trends in Cognitive Sciences, 7(9), 415–423. doi:10.1016/S1364-6613(03)00197-9
  • Daunizeau, J., Friston, K. J., & Kiebel, S. J. (2009). Variational Bayesian identification and prediction of stochastic nonlinear dynamic causal models. Physica D: Nonlinear Phenomena, 238(21), 2089–2118. doi:10.1016/j.physd.2009.08.002
  • Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69(6), 1204–1215. doi:10.1016/j.neuron.2011.02.027
  • Deelman, B., Maring, W., & Otten, V. (1989). De CST, een gestandaardiseerde screeningsmethode voor dementie. In J. Schroots, A. Bouma, G. Braam, A. Groeneveld, D. Ringoir, & C. Tempelman (Eds.), Gezond zijn is ouder worden (pp. 163–170). Assen: Van Gorcum.
  • Di Martino, A., Scheres, A., Margulies, D. S., Kelly, A. M. C., Uddin, L. Q., Shehzad, Z., … Milham, M. P. (2008). Functional connectivity of human striatum: A resting state FMRI study. Cerebral Cortex, 18(12), 2735–2747. doi:10.1093/cercor/bhn041
  • Dickinson, A. (1985). Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society B: Biological Sciences, 308(1135), 67–78. doi:10.1098/rstb.1985.0010
  • Doll, B. B., Simon, D. A., & Daw, N. D. (2012). The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology, 22(6), 1075–1081. doi:10.1016/J.CONB.2012.08.003
  • Eppinger, B., Hämmerer, D., & Li, S.-C. (2011). Neuromodulation of reward-based learning and decision making in human aging. Annals of the New York Academy of Sciences, 1235, 1–17. doi:10.1111/j.1749-6632.2011.06230.x
  • Eppinger, B., & Kray, J. (2011). To choose or to avoid: Age differences in learning from positive and negative feedback. Journal of Cognitive Neuroscience, 23(1), 41–52. doi:10.1162/jocn.2009.21364
  • Eppinger, B., Schuck, N. W., Nystrom, L. E., & Cohen, J. D. (2013). Reduced striatal responses to reward prediction errors in older compared with younger adults. The Journal of Neuroscience, 33(24), 9905–9912. doi:10.1523/JNEUROSCI.2942-12.2013
  • Frank, M. J. (2005). Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience, 17(1), 51–72. doi:10.1162/0898929052880093
  • Frank, M. J., Seeberger, L. C., & O’reilly, R. C. (2004). By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science, 306(5703), 1940–1943. doi:10.1126/science.1102941
  • Gershman, S. J., & Daw, N. D. (2017). Reinforcement learning and episodic memory in humans and animals: An integrative framework. Annual Review of Psychology, 68, 101–128. doi:10.1146/annurev-psych-122414-033625
  • Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences, 108(Supplement_3), 15647–15654. doi:10.1073/pnas.1014269108
  • Grieve, S. M., Williams, L. M., Paul, R. H., Clark, C. R., & Gordon, E. (2007). Cognitive aging, executive function, and fractional anisotropy: A diffusion tensor MR imaging study. American Journal of Neuroradiology, 28(2), 226–235. Retrieved from http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=17296985
  • Haber, S. N., & Knutson, B. (2010). The reward circuit: Linking primate anatomy and human imaging. Neuropsychopharmacology, 35(1), 4–26. doi:10.1038/npp.2009.129
  • Hämmerer, D., & Eppinger, B. (2012). Dopaminergic and prefrontal contributions to reward-based learning and outcome monitoring during child development and aging. Developmental Psychology, 48(3), 862–874. doi:10.1037/a0027342
  • Hämmerer, D., Li, S.-C., Müller, V., & Lindenberger, U. (2011). Life span differences in electrophysiological correlates of monitoring gains and losses during probabilistic reinforcement learning. Journal of Cognitive Neuroscience, 23(3), 579–592. doi:10.1162/jocn.2010.21475
  • Haruno, M., & Kawato, M. (2006). Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: FMRI examination in stimulus-action-reward association learning. Neural Networks, 19(8), 1242–1254. doi:10.1016/j.neunet.2006.06.007
  • Head, D., Kennedy, K. M., Rodrigue, K. M., & Raz, N. (2009). Age differences in perseveration: Cognitive and neuroanatomical mediators of performance on the Wisconsin card sorting test. Neuropsychologia, 47(4), 1200–1203. doi:10.1016/J.NEUROPSYCHOLOGIA.2009.01.003
  • Karrer, T. M., Josef, A. K., Mata, R., Morris, E. D., & Samanez-Larkin, G. R. (2017). Reduced dopamine receptors and transporters but not synthesis capacity in normal aging adults: A meta-analysis. Neurobiology of Aging, 57, 36–46. doi:10.1016/j.neurobiolaging.2017.05.006
  • Ligneul, R. (2019). Sequential exploration in the Iowa gambling task: Validation of a new computational model in a large dataset of young and old healthy participants. PLoS Computational Biology, 15(6), e1006989. doi:10.1371/journal.pcbi.1006989
  • Logan, J. M., & Balota, D. A. (2008). Expanded vs. equal interval spaced retrieval practice: Exploring different schedules of spacing and retention interval in younger and older adults. Aging, Neuropsychology, and Cognition, 15(3), 257–280. doi:10.1080/13825580701322171
  • Ludvig, E. A., Madan, C. R., McMillan, N., Xu, Y., & Spetch, M. L. (2018). Living near the edge: How extreme outcomes and their neighbors drive risky choice. Journal of Experimental Psychology: General, 147(12), 1905–1918. doi:10.1037/xge0000414
  • Madden, D. J., Costello, M. C., Dennis, N. A., Davis, S. W., Shepler, A. M., Spaniol, J., … Cabeza, R. (2010). Adult age differences in functional connectivity during executive control. Neuroimage, 52(2), 643–657. doi:10.1016/j.neuroimage.2010.04.249
  • Maddox, G. B., Balota, D. A., Coane, J. H., & Duchek, J. M. (2011). The role of forgetting rate in producing a benefit of expanded over equal spaced retrieval in young and older adults. Psychology and Aging, 26(3), 661–670. doi:10.1037/a0022942
  • Mell, T., Heekeren, H. R., Marschner, A., Wartenburger, I., Villringer, A., & Reischies, F. M. (2005). Effect of aging on stimulus-reward association learning. Neuropsychologia, 43(4), 554–563. doi:10.1016/j.neuropsychologia.2004.07.010
  • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. doi:10.1037/h0043158
  • Packard, M. G., & Knowlton, B. J. (2002). Learning and memory functions of the Basal Ganglia. Annual Review of Neuroscience, 25, 563–593. doi:10.1146/annurev.neuro.25.112701.142937
  • Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442(7106), 1042–1045. doi:10.1038/nature05051
  • Ponds, R. W. H. M., Verhey, F. R. J., Rozendaal, N., Jolles, J., & Deelman, B. G. (1992). Brief cognitive screening tests for dementia: Comparison of the mini-mental state examination and the cognitive screening tests. In H. Bouma & J. A. M. Graafmans (Eds.), Gerontechnology (pp. 261–264). Retrieved from http://www.narcis.nl/publication/RecordID/oai:dare:8410
  • Radulescu, A., Daniel, R., & Niv, Y. (2016). The effects of aging on the interaction between reinforcement learning and attention. Psychology and Aging, 31(7), 747–757. doi:10.1037/pag0000112
  • Raz, N., Lindenberger, U., Rodrigue, K. M., Kennedy, K. M., Head, D., Williamson, A., … Acker, J. D. (2005). Regional brain changes in aging healthy adults: General trends, individual differences and modifiers. Cerebral Cortex, 15(11), 1676–1689. Retrieved from https://doi.org/10.1093/cercor/bhi044
  • Raz, N., Rodrigue, K. M., Kennedy, K. M., Head, D., Gunning-Dixon, F., & Acker, J. D. (2003). Differential aging of the human striatum: Longitudinal evidence. 24(9), 1849–1856. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/14561615
  • Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). New York, NY: Appleton-Century-Crofts. Retrieved from http://www.ualberta.ca/~egray/teaching/Rescorla&Wagner1972.pdf
  • Ridderinkhof, K. R., Span, M. M., & van der Molen, M. W. (2002). Perseverative behavior and adaptive control in older adults: Performance monitoring, rule induction, and set shifting. Brain and Cognition, 49(3), 382–401. Retrieved from https://doi.org/10.1006/brcg.2001.1506
  • Rouhani, N., Norman, K. A., & Niv, Y. (2018). Dissociable effects of surprising rewards on learning and memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(9), 1430–1443. doi:10.1037/xlm0000518
  • Rutledge, R. B., Lazzaro, S. C., Lau, B., Myers, C. E., Gluck, M. A., & Glimcher, P. W. (2009). Dopaminergic drugs modulate learning rates and perseveration in Parkinson’s patients in a dynamic foraging task. Journal of Neuroscience, 29(48), 15104–15114. doi:10.1523/JNEUROSCI.3524-09.2009
  • Rypma, B., Prabhakaran, V., Desmond, J. E., Glover, G. H., & Gabrieli, J. D. (1999). Load-dependent roles of frontal brain regions in the maintenance of working memory. NeuroImage, 9(2), 216–226. doi:10.1006/nimg.1998.0404
  • Salat, D. H., Greve, D. N., Pacheco, J. L., Quinn, B. T., Helmer, K. G., Buckner, R. L., & Fischl, B. (2009). Regional white matter volume differences in nondemented aging and Alzheimer’s disease. NeuroImage, 44(4), 1247–1258. Retrieved from https://doi.org/10.1016/j.neuroimage.2008.10.030
  • Schmand, B. A., Bakker, D., Saan, R. J., & Louman, J. (1991). De Nederlandse leestest voor Volwassenen: Een maat voor het premorbide intelligentieniveau/The Dutch adult reading test: A measure of premorbid intelligence. Tijdschrift Voor Gerontologie En Geriatrie, 22(1), 15–19.
  • Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. doi:10.1126/science.275.5306.1593
  • Schultz, W. (2013). Updating dopamine reward signals. Current Opinion in Neurobiology, 23(2), 229–238. doi:10.1016/j.conb.2012.11.012
  • Seger, C. A., & Cincotta, C. M. (2006). Dynamics of frontal, striatal, and hippocampal systems during rule learning. Cerebral Cortex, 16(11), 1546–1555. doi:10.1093/cercor/bhj092
  • Simon, H. A. (1974). How big is a chunk? By combining data from several experiments, a basic human memory unit can be identified and measured. Science, 183(4124), 482–488. doi:10.1126/science.183.4124.482
  • Simon, J. R., Howard, J. H., & Howard, D. V. (2010). Adult age differences in learning from positive and negative probabilistic feedback. Neuropsychology, 24(4), 534–541. doi:10.1037/a0018652
  • Simone, P. M., Bell, M. C., & Cepeda, N. J. (2013). Diminished but not forgotten: Effects of aging on magnitude of spacing effect benefits. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 68(5), 674–680. doi:10.1093/geronb/gbs096
  • Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology, 6(2), 174–215. http://dx.doi.org/10.1037/0033-2909.87.2.245.
  • Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2), 245–251. Retrieved from http://dx.doi.org/10.1037/0033-2909.87.2.245
  • Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. xavierdupre.fr. Cambridge, MA: MIT Press. Retrieved from http://www.xavierdupre.fr/enseignement/projet_data/apprentissage_renforcement_RL-3.pdf
  • Tisserand, D. J., van Boxtel, M. P. J., Pruessner, J. C., Hofman, P., Evans, A. C., & Jolles, J. (2004). A voxel-based morphometric study to determine individual differences in gray matter density associated with age and cognitive change over time. Cerebral Cortex, 14(9), 966–973. doi:10.1093/cercor/bhh057
  • Toppino, T. C., & Gerbier, E. (2014). About practice: Repetition, spacing, and abstraction. Psychology of Learning and Motivation, 60, 113–189. doi:10.1016/B978-0-12-800090-8.00004-4
  • Torrubia, R., Ávila, C., Moltó, J., & Caseras, X. (2001). The Sensitivity to Punishment and Sensitivity to Reward Questionnaire (SPSRQ) as a measure of Gray’s anxiety and impulsivity dimensions. Personality and Individual Differences, 31(6), 837–862. doi:10.1016/S0191-8869(00)00183-5
  • Turner, M. L., & Engle, R. W. (1989). Is working memory capacity task dependent? Journal of Memory and Language, 28(2), 127–154. doi:10.1016/0749-596X(89)90040-5
  • Unsworth, N., Fukuda, K., Awh, E., & Vogel, E. K. (2014). Working memory and fluid intelligence: Capacity, attention control, and secondary memory retrieval. Cognitive Psychology, 71, 1–26. doi:10.1016/J.COGPSYCH.2014.01.003
  • van de Vijver, I., Cohen, M. X, & Ridderinkhof, K. R. (2014). Aging affects medial but not anterior frontal learning-related theta oscillations. Neurobiology of Aging, 35(3), 692–704 https://doi.org/10.1016/j.neurobiolaging.2013.09.006.
  • van de Vijver, I., Ridderinkhof, K. R., Harsay, H., Reneman, L., Cavanagh, J. F., Buitenweg, J. I. V., & Cohen, M. X (2016). Frontostriatal anatomical connections predict age- and difficulty-related differences in reinforcement learning. Neurobiology of Aging, 46, 1–12. doi:10.1016/j.neurobiolaging.2016.06.002
  • van de Vijver, I., Ridderinkhof, K. R., & de Wit, S. (2015). Age-related changes in deterministic learning from valenced performance feedback. Aging, Neuropsychology, and Cognition, 22(5), 595–619. https://doi.org/10.1080/13825585.2015.1020917 .
  • Wechsler, D. (2008). Wechsler adult intelligence scale (4th ed.). San Antonio, TX: Pearson Assessment.
  • Weiler, J. A., Bellebaum, C., & Daum, I. (2008). Aging affects acquisition and reversal of reward-based associative learning. Learning & Memory, 15(4), 190–197. doi:10.1111/j.1460-9568.2012.08017.x
  • Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012). Generalization of value in reinforcement learning by humans. European Journal of Neuroscience, 35(7), 1092–1104. doi:10.1111/j.1460-9568.2012.08017.x
  • Wimmer, G. E., & Shohamy, D. (2012). Preference by association: How memory mechanisms in the hippocampus bias decisions. Science, 338(6104), 270–273. doi:10.1126/science.1223252
  • Wittmann, B. C., Schott, B. H., Guderian, S., Frey, J. U., Heinze, H.-J., & Düzel, E. (2005). Reward-related fMRI activation of dopaminergic midbrain is associated with enhanced hippocampus- Dependent long-term memory formation. Neuron, 45(3), 459–467. doi:10.1016/J.NEURON.2005.01.010
  • Yesavage, J. A., Brink, T. L., Rose, T. L., Lum, O., Huang, V., Adey, M., & Leirer, V. O. (1982). Development and validation of a geriatric depression screening scale: A preliminary report. Journal of Psychiatric Research, 17(1), 37–49. doi:10.1016/0022-3956(82)90033-4
  • Ziegler, D. A., Piguet, O., Salat, D. H., Prince, K., Connally, E., & Corkin, S. (2010). Cognition in healthy aging is related to regional white matter integrity, but not cortical thickness. Neurobiology of Aging, 31(11), 1912–1926. Retrieved from https://doi.org/10.1016/j.neurobiolaging.2008.10.015