A reinforcement learning model of joy, distress, hope and fear

References

• Baumeister, R. F., Vohs, K. D., DeWall, C. N., & Zhang, L. (2007). How emotion shapes behavior: Feedback, anticipation, and reflection, rather than direct causation. Personality and Social Psychology Review, 11(2), 167–203. doi: 10.1177/1088868307301033
   PubMed Web of Science ®Google Scholar
• Berridge, K. C. (2003). Pleasures of the brain. Brain and Cognition, 52(1), 106–128. doi: 10.1016/S0278-2626(03)00014-9
   PubMed Web of Science ®Google Scholar
• Berridge, K. C., & Robinson, T. E. (2003). Parsing reward. Trends in Neurosciences, 26(9), 507–513. doi: 10.1016/S0166-2236(03)00233-9
   PubMed Web of Science ®Google Scholar
• Berridge, K. C., Robinson, T. E., & Aldridge, J. W. (2009). Dissecting components of reward: “liking”, “wanting”, and learning. Current Opinion in Pharmacology, 9(1), 65–73. doi: 10.1016/j.coph.2008.12.014
   PubMed Web of Science ®Google Scholar
• Bogacz, R., & Gurney, K. (2007). The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Computation, 19(2), 442–477. doi: 10.1162/neco.2007.19.2.442
   PubMed Web of Science ®Google Scholar
• Bottan, N. L., & Perez Truglia, R. (2011). Deconstructing the hedonic treadmill: Is happiness autoregressive? Journal of Socio-Economics, 40(3), 224–236. doi: 10.1016/j.socec.2011.01.007
   Google Scholar
• Brickman, P., Coates, D., & Janoff-Bulman, R. (1978). Lottery winners and accident victims: Is happiness relative? Journal of Personality and Social Psychology, 36(8), 917–927. doi: 10.1037/0022-3514.36.8.917
   PubMed Web of Science ®Google Scholar
• Broekens, J. (2007). Affect and learning: A computational analysis (PhD thesis). Leiden University.
   Google Scholar
• Broekens, J., & Baarslag, T. (2014). Optimistic risk perception in the temporal difference error explains the relation between risk-taking, gambling, sensation-seeking and low fear. Retrieved from arXiv:1404.2078.
   Google Scholar
• Broekens, J., Kosters, W., & Verbeek, F. (2007). On affect and self-adaptation: Potential benefits of valence-controlled action-selection. In J. Mira & J. A. lvarez (Eds.), Bio-inspired modeling of cognitive tasks (pp. 357–366). Berlin: Springer.
   Google Scholar
• Broekens, J., Marsella, S., & Bosse, T. (2013). Challenges in computational modeling of affective processes. IEEE Transactions on Affective Computing, 4(3), 242–245. doi: 10.1109/T-AFFC.2013.23
   Google Scholar
• Bryson, J. J. (2007). Mechanisms of action selection: Introduction to the special issue. Adaptive Behavior – Animals, Animats, Software Agents, Robots, Adaptive Systems, 15(1), 5–8.
   Google Scholar
• Cañamero, L. (2005). Emotion understanding from the perspective of autonomous robots research. Neural Networks, 18(4), 445–455. doi: 10.1016/j.neunet.2005.03.003
   PubMed Web of Science ®Google Scholar
• Carver, C. S., Scheier, M. F., & Segerstrom, S. C. (2010). Optimism. Clinical Psychology Review, 30(7), 879–889. doi: 10.1016/j.cpr.2010.01.006
   PubMed Web of Science ®Google Scholar
• Cos, I., Canamero, L., Hayes, G. M., & Gillies, A. (2013). Hedonic value: Enhancing adaptation for motivated agents. Adaptive Behavior, 21(6), 465–483. doi: 10.1177/1059712313486817
   Web of Science ®Google Scholar
• Critchfield, T. S., Paletz, E. M., MacAleese, K. R., & Newland, M. C. (2003). Punishment in human choice: Direct or competitive suppression? Journal of the Experimental Analysis of Behavior, 80(1), 1–27. doi: 10.1901/jeab.2003.80-1
   PubMed Web of Science ®Google Scholar
• Damasio, A. R. (1994). Descartes' error: Emotion reason and the human brain. New York: Penguin Putnam.
   Google Scholar
• Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36(2), 285–298. doi: 10.1016/S0896-6273(02)00963-7
   PubMed Web of Science ®Google Scholar
• Dias, J., Mascarenhas, S., & Paiva, A. (2011). Fatima modular: Towards an agent architecture with a generic appraisal framework. Proceedings of the workshop on standards in emotion modeling, Leiden.
   Google Scholar
• Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15(4–6), 495–506. doi: 10.1016/S0893-6080(02)00044-8
   PubMed Web of Science ®Google Scholar
• El-Nasr, M. S., Yen, J., & Ioerger, T. R. (2000). Flame: Fuzzy logic adaptive model of emotions. Autonomous Agents and Multi-agent Systems, 3(3), 219–257. doi: 10.1023/A:1010030809960
   Web of Science ®Google Scholar
• Foa, E. B., & Kozak, M. J. (1986). Emotional processing of fear: Exposure to corrective information. Psychological Bulletin, 99(1), 20–35. doi: 10.1037/0033-2909.99.1.20
   PubMed Web of Science ®Google Scholar
• Frijda, N. H. (2004). Emotions and action. In A. S. R. Manstead & N. H. Frijda (Eds.), Feelings and emotions: The Amsterdam symposium (pp. 158–173). Cambridge: Cambridge University Press (Feelings and emotions: The Amsterdam symposium).
   Google Scholar
• Frijda, N. H., Kuipers, P., & ter Schure E. (1989). Relations among emotion, appraisal, and emotional action readiness. Journal of Personality and Social Psychology, 57(2), 212–228. doi: 10.1037/0022-3514.57.2.212
   Web of Science ®Google Scholar
• Gadanho, S. C. (1999). Reinforcement learning in autonomous robots: An empirical investigation of the role of emotions (Unpublished doctoral dissertation). University of Edinburgh.
   Google Scholar
• Gratch, J., Marsella, S., Wang, N., & Stankovic, B. (2009). Assessing the validity of appraisal-based models of emotion. 3rd International conference on affective computing and intelligent interaction and workshops, 2009, ACII 2009, Amsterdam.
   Google Scholar
• Haruno, M., & Kawato, M. (2006). Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. Journal of Neurophysiology, 95(2), 948–959. doi: 10.1152/jn.00382.2005
   PubMed Web of Science ®Google Scholar
• Hogewoning, E., Broekens, J., Eggermont, J., & Bovenkamp, E. (2007). Strategies for affect-controlled action-selection in soar-rl. In J. Mira & J. R. Álvarez (Eds.), Nature inspired problem-solving methods in knowledge engineering: Lecture Notes in Computer Science (LNCS vol. 4528, pp. 501–510). Berlin: Springer.
   Google Scholar
• Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109(4), 679–709. doi: 10.1037/0033-295X.109.4.679
   PubMed Web of Science ®Google Scholar
• Houk, J., Bastianen, C., Fansler, D., Fishbach, A., Fraser, D., Reber, P., …, Simo, L. (2007). Action selection and refinement in subcortical loops through basal ganglia and cerebellum. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1485), 1573–1583. doi: 10.1098/rstb.2007.2063
   PubMed Web of Science ®Google Scholar
• Jacobs, E., Broekens, J., & Jonker, C. (2014a). Emergent dynamics of joy, distress, hope and fear in reinforcement learning agents. In S. Barrett, S. Devlin, & D. Hennes (Eds.), Adaptive learning agents workshop at aamas 2014, Paris.
   Google Scholar
• Jacobs, E., Broekens, J., & Jonker, C. (2014b). Joy, distress, hope and fear in reinforcement learning. In A. Bazzan, M. Huhns, A. Lomuscio, & P. Scerri (Eds.), Proceedings of the 2014 international conference on autonomous agents and multi-agent systems (pp. 1615–1616). Paris: International Foundation for Autonomous Agents and Multiagent Systems.
   Google Scholar
• Kiryazov, K., Lowe, R., Becker-Asano, C., & Ziemke, T. (2011). Modelling embodied appraisal in humanoids: Grounding PAD space for augmented autonomy. In: Proceedings of the Workshop on Standards in Emotion Modelling, Leiden.
   Google Scholar
• Krichmar, J. L. (2008). The neuromodulatory system: A framework for survival and adaptive behavior in a challenging world. Adaptive Behavior, 16(6), 385–399. doi: 10.1177/1059712308095775
   Web of Science ®Google Scholar
• Lahnstein, M. (2005). The emotive episode is a composition of anticipatory and reactive evaluations. In L. Ca namero (Ed.), Agents that want and like: Motivational and emotional roots of cognition and action. Papers from the AISB'05 symposium (pp. 62–69). Hatfield: AISB Press.
   Google Scholar
• Lewis, M. D. (2005). Bridging emotion theory and neurobiology through dynamic systems modeling. Behavioral and Brain Sciences, 28(2), 169–194.
   PubMed Web of Science ®Google Scholar
• Marinier, R., & Laird, J. E. (2008). Emotion-driven reinforcement learning. In Cognitive science (pp. 115–120). Washington, DC: Cognitive Science Society.
   Google Scholar
• Marsella S. C., & Gratch, J. (2009). Ema: A process model of appraisal dynamics. Cognitive Systems Research, 10(1), 70–90. doi: 10.1016/j.cogsys.2008.03.005
   Web of Science ®Google Scholar
• Marsella, S., Gratch, J., & Petta, P. (2010). Computational models of emotion. In K. R. Scherer, T. Bänziger, & E. Roesch (Eds.), A blueprint for affective computing (pp. 21–45). Oxford: Oxford University Press..
   Google Scholar
• Montague, P. R., King-Casas, B., & Cohen, J. D. (2006). Imaging valuation models in human choice. Annual Review of Neuroscience, 29, 417–448. doi: 10.1146/annurev.neuro.29.051605.112903
   PubMed Web of Science ®Google Scholar
• Moren, J., & Balkenius, C. (2000). A computational model of emotional learning in the amygdala. In J. A. Meyer, A. Berthoz, D. Floreano, H. L. Roitblat, & S. W. Wilson (Eds.), From animals to animats 6. Proceedings of the 6th International Conference on the Simulation of Adaptive Behaviour (vol. 6, pp. 115–124). Cambridge, MA: MIT Press.
   Google Scholar
• Myers, K. M., & Davis, M. (2006). Mechanisms of fear extinction. Molecular Psychiatry, 12(2), 120–150. doi: 10.1038/sj.mp.4001939
   PubMed Web of Science ®Google Scholar
• O'Doherty, J. P. (2004). Reward representations and reward-related learning in the human brain: Insights from neuroimaging. Current Opinion in Neurobiology, 14(6), 769–776. doi: 10.1016/j.conb.2004.10.016
   PubMed Web of Science ®Google Scholar
• Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. Cambridge: Cambridge University Press.
   Google Scholar
• Prescott, T. J., Bryson, J. J., & Seth, A. K. (2007). Introduction. Modelling natural action selection. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1485), 1521–1529. doi: 10.1098/rstb.2007.2050
   PubMed Web of Science ®Google Scholar
• Redish A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007). Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review, 114(3), 784–805. doi: 10.1037/0033-295X.114.3.784
   PubMed Web of Science ®Google Scholar
• Reisenzein, R. (2009). Emotional experience in the computational belief-desire theory of emotion. Emotion Review, 1(3), 214–222. doi: 10.1177/1754073909103589
   Web of Science ®Google Scholar
• Robinson, P., & el Kaliouby, R. (2009). Computation of emotions in man and machines. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3441–3447. doi: 10.1098/rstb.2009.0198
   PubMed Web of Science ®Google Scholar
• Rolls, E. T. (2000). Precis of the brain and emotion. Behavioral and Brain Sciences, 20, 177–234. doi: 10.1017/S0140525X00002429
   Web of Science ®Google Scholar
• Rolls, E. T., & Grabenhorst, F. (2008). The orbitofrontal cortex and beyond: From affect to decision-making. Progress in Neurobiology, 86(3), 216–244. doi: 10.1016/j.pneurobio.2008.09.001
   PubMed Web of Science ®Google Scholar
• Schaffer, H. (1974). Cognitive components of the infant's response to strangeness. In M. Lewis, & L. Rosenblum (Eds.), The origins of fear. New York: Wiley.
   Google Scholar
• Scherer, K. R. (2001). Appraisal considered as a process of multilevel sequential checking. In K. R. Scherer, A. Schorr & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 92–120). Oxford: Oxford University Press.
   Google Scholar
• Scherer, K. R. (2009). Emotions are emergent processes: They require a dynamic computational architecture. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3459–3474. doi: 10.1098/rstb.2009.0141
   PubMed Web of Science ®Google Scholar
• Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1–27.
   PubMed Web of Science ®Google Scholar
• Schweighofer, N., & Doya, K. (2003). Meta-learning in reinforcement learning. Neural Networks, 16(1), 5–9. doi: 10.1016/S0893-6080(02)00228-9
   PubMed Web of Science ®Google Scholar
• Sequeira, P. (2013a). Socio-emotional reward design for intrinsically motivated learning agents (Unpublished doctoral dissertation). Universidada Técnica de Lisboa.
   Google Scholar
• Sequeira, P. (2013b). Socio-emotional reward design for intrinsically motivated learning agents (Unpublished doctoral dissertation). Universidade Tecnica de Lisboa.
   Google Scholar
• Sequeira, P., Melo, F., & Paiva, A. (2011). Emotion-based intrinsic motivation for reinforcement learning agents. In S. D'Mello, A. Graesser, B. Schuller, & J. C. Martin (Eds.), Affective computing and intelligent interaction (vol. 6974, pp. 326–336). Berlin, Heidelberg: Springer.
   Google Scholar
• Sequeira, P., Melo, F., & Paiva, A. (in press). Emergence of emotional appraisal signals in reinforcement learning agents. Autonomous Agents and Multi-Agent Systems.
   PubMed Web of Science ®Google Scholar
• Simon, H. A. (1955). A behavioral model of rational choice. The Quarterly Journal of Economics, 69(1), 99–118. doi: 10.2307/1884852
   Web of Science ®Google Scholar
• Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129–138. doi: 10.1037/h0042769
   PubMed Web of Science ®Google Scholar
• Snyder, C. R. (2002). Target article: Hope theory: Rainbows in the mind. Psychological Inquiry, 13(4), 249–275. doi: 10.1207/S15327965PLI1304_01
   Web of Science ®Google Scholar
• Sprott, J. (2005). Dynamical models of happiness. Nonlinear Dynamics, Psychology, and Life Sciences, 9(1), 23–36.
   PubMedGoogle Scholar
• Sroufe, L. A. (1997). Emotional development: The organization of emotional life in the early years. Cambridge: Cambridge University Press.
   Google Scholar
• Steephen, J. E. (2013). HED: A computational model of affective adaptation and emotion dynamics. IEEE Transactions on Affective Computing, 4(2), 197–210. doi: 10.1109/T-AFFC.2013.2
   Web of Science ®Google Scholar
• Suri, R. E. (2002). Td models of reward predictive responses in dopamine neurons. Neural Networks, 15(4–6), 523–533. doi: 10.1016/S0893-6080(02)00046-1
   PubMed Web of Science ®Google Scholar
• Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (vol. 1). Cambridge: Cambridge University Press.
   Google Scholar
• Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004). Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience, 7(8), 887–893. doi: 10.1038/nn1279
   PubMed Web of Science ®Google Scholar
• Veenhoven, R. (1991). Is happiness relative? Social Indicators Research, 24(1), 1–34. doi: 10.1007/BF00292648
   Web of Science ®Google Scholar
• Zajonc, R. B., & Markus, H. (1982). Affective and cognitive factors in preferences. Journal of Consumer Research, 9(2), 123–131. doi: 10.1086/208905
   Web of Science ®Google Scholar