Multi-step planning with learned effects of partial action executions

Abstract

In this paper, we propose a novel affordance model, which combines object, action, and effect information in the latent space of a predictive neural network architecture that is built on Conditional Neural Processes. Our model allows us to make predictions of intermediate effects expected to be obtained during action executions and make multi-step plans that include partial actions. We first compared the prediction capability of our model using an existing interaction data set and showed that it outperforms a recurrent neural network-based model in predicting the effects of lever-up actions. Next, we showed that our model can generate accurate effect predictions for other actions, such as push and grasp actions. Our system was shown to generate successful multi-step plans to bring objects to desired positions using the traditional A* search algorithm. Furthermore, we realized a continuous planning method and showed that the proposed system generated more accurate and effective plans with sequences of partial action executions compared to plans that only consider full action executions using both planning algorithms.

GRAPHICAL ABSTRACT

Keywords:

• Affordances
• effect prediction
• object motion trajectory prediction
• multi-step planning

Acknowledgements

The numerical calculations reported in this paper were partially performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources). The authors would like to thank Alper Ahmetoglu for providing insightful comments for this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was supported by TUBITAK (The Scientific and Technological Research Council of Turkey) ARDEB; 1001 program (project number: 120E274); TUBITAK BIDEB; 2210-A program; and by the BAGEP Award of the Science Academy.

Notes on contributors

Hakan Aktas

Hakan Aktas is currently a senior computer engineering student at Bogazici University and an undergraduate researcher at the Cognitive Learning and Robotics (Colors) Lab of Bogazici University. His interests lie in robot learning in general, more specifically cognitive robotics, cross-embodiment learning, and imitation learning.

Utku Bozdogan

Utku Bozdogan was a Master's in Computer Engineering student who graduated in 2022. He is currently a machine learning engineer at Venara Ventures, Istanbul, Turkey.

Emre Ugur

Emre Ugur is currently serving as an Associate Professor in the Department of Computer Engineering, and he is the head of the Cognitive Science Master's Program and the Cognitive, Learning, and Robotics (CoLoRs) laboratory. He received his Ph.D. in Computer Engineering from Middle East Technical University, Turkey. He has served as a research scientist at ATR in Japan, worked as an Adjunct Associate Professor at Osaka University, and worked as a senior researcher at Innsbruck University. He has been a Principle Investigator in several projects funded by the European Union, TUBITAK, and JST. His interests lie in robotics, robot learning, cognitive robotics, and cognitive sciences.