A knowledge-driven layered inverse reinforcement learning approach for recognizing human intents

ABSTRACT

There is a rising trend in exploring the capability of inverse reinforcement learning (IRL) in high dimensional demonstrations. Our aim is to recognise human intents from video data within an IRL framework. For this, we present a two-layered maximum likelihood IRL model. The usefulness of knowledge representation (KR) schemes and availability of advisors at different layers is exploited through this model. Two main aspects are addressed: a. the importance of having abstract high-level information to the IRL framework in terms of semantic object affordance and b. deductively exploring the utility of a state at different temporal abstractions. The effectiveness of the proposed model has been evaluated with the help of standard Cornell Activity Dataset (CAD-120).

KEYWORDS:

• Inverse reinforcement learning
• knowledge representation
• advisor
• object affordance

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. We have used Qualitative Distance Calculus (QDC) in this reported work. QDC proposed by Clementini, Di Felice, and Hernández (1997) is a qualitative relational calculus which expresses the qualitative Euclidean distance between two points depending on defined region boundaries.

2. Properties of objects (appeared in the video demonstrations) have been inferred from Object Property Ontology (O-PrO) (Bhattacharyya et al., 2017). Non-trivial object properties represented in O-PrO and its usefulness in video-based evaluation platform help us to consider this knowledge structure in our proposed model.

3. Both abstract MDP and CAMDP are the traditional MDP solver; together with a knowledge level inserted within it. It helps to work on less number of states and actions.

4. see Appendix A.

5. Readers may refer to Appendix A and Appendix B to know the details about this procedure.

6. (MLN) (Richardson & Domingos, 2006) is a Statistical Relational Learning (SRL) scheme. It can also be considered as a knowledge representation language which can combine symbolic information from a household domain (background) knowledge and information generated by processing the videos.

7. There are reasons for utilising MLN in our proposed knowledge-based approach. MLN generalises over the existing probabilistic models, including hidden Bayesian networks, Markov models, and stochastic grammars. In addition to being a probabilistic framework, MLN provides the ability to write more flexible rules with existential quantifiers over sets of entities. In terms of expressive power, MLN is better as compared to other probabilistic rule-based methods such as dynamic Bayesian networks or attribute grammars (Tran & Davis, 2008).

8. Readers may refer to Appendix 8 to know the details about this computational step as well as the LMLIRL model.

9. See Appendix B for details.

10. R(S) denotes learnt reward function by top-most layer agent; while r(s) is for bottom-most layer agent.

11. See Algorithm 1 (step 14 and step 17) utilised for trajectory segmentation. Further description about the estimation of qualitative relations can be obtained in Appendix A.

12. https://alchemy.cs.washington.edu/.

13. This repository was also written in Java.

14. You may visit https://github.com/RupamBhattacharyya/CAD120-Object-Affordances to know about rule $r^1$ and the procedure to detect activated object affordance.