Robot Skill Generalization via KIS-GMM

Abstract

A long-standing challenge for a robotic manipulation system operating in real-world scenarios is adapting and generalizing its acquired motor skills to unseen environments. We tackle this challenge employing hybrid skill models that integrate imitation and reinforcement paradigms, to explore how the learning and adaptation of a skill, along with its core grounding in the scene through a learned keypoint, can facilitate such generalization. To that end, we develop Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models (KIS-GMM) approach that learns to predict the reference of a dynamical system within the scene as a 3D keypoint, leveraging visual observations obtained by the robot’s physical interactions during skill learning. Through conducting comprehensive evaluations in both simulated and real-world environments, we show that our method enables a robot to gain a significant zero-shot generalization to novel environments and to refine skills in the target environments faster than learning from scratch. Importantly, this is achieved without the need for new ground truth data. Moreover, our method effectively copes with scene displacements.

Figure 1. Overview of robot skill generalization via KIS-GMM.

Approach

We introduce the Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models (KIS-GMM), a robot learning framework specifically crafted for fostering adaptive and generalizable robot skills. This framework is segmented into two distinct phases:

1. In the first phase, a robot skill is acquired by learning a parameterized dynamical system from a few demonstrations within a single source environment. This stage masters the skill not only by adapting dynamical system parameters through physical interactions, high-dimensional sensory data, and sparse rewards but also by deriving a pivotal keypoint crucial for grounding the skill in the scene.

2. In the second phase, the dynamical system is generalized to unseen environments that share structural similarities to the source environment by transferring the learned keypoint, conducting any necessary refinements to either the skill policy or keypoint detection if required.

Figure 2: KIS-GMM learns to adapt a robot skill and generalize it to unseen environments. Adaptation is achieved through parameter refinements (Δ𝜃) utilizing physical interactions and sparse rewards. Generalization is facilitated by transferring a learned keypoint (𝜉ˆ∗) to novel environments.

Experiments

In the real-world experiment, we examine Door Opening and Drawer Opening skills. First we pursue to master learned skills in a source environment

Open Door Skill in the Source Environment A

Open Drawer Skill in the Source Environment A

Figure 3. During the skill refinement phase within the source environment, the Key agent utilizes the wrist camera’s observations to predict the 3D position of the dynamical system’s reference point in the frame of the wrist camera

The baseline skill models, lacking awareness of the dynamical system’s reference point in target environments, struggle to anchor the skill to these new contexts, leading to an inability to generalize the learned skill effectively.

Target Environment B

Target Environment C

Target Environment D

Figure 4. GMM and SAC-GMM models are not capable of grounding the skill’s dynamical system in novel scenes.

KIS-GMM learns to identify a 3D keypoint representation that grounds the skill’s dynamical system in novel scenes. Below you see that our model employs the wrist camera observation to infer a keypoint surrounding door and drawer handles in unseen environements.

Figure 5. Our keypoint detector leverages the wrist camera observations to predict the dynamical system’s reference point.

KIS-GMM successfully opens four visually distinct doors and drawers in the real world, having been initially trained on just one of each, respectively.

Target Environment B

Target Environment C

Target Environment D

Figure 6. KIS-GMM generalizes to unseen doors and drawers.

Publications

Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models
Iman Nematollahi^*, Kirill Yankov^*, Wolfram Burgard, Tim Welschehold
arxiv
Pdf

Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models Iman Nematollahi^*, Kirill Yankov^*, Wolfram Burgard, Tim Welschehold arxiv Pdf

BibTeX citation

                
@inproceedings{nematollahi23iser,
author  = {Iman Nematollahi and Kirill Yankov and Wolfram Burgard and Tim Welschehold}
title   = {Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models},
booktitle = {Proceedings of the International Symposium on Experimental Robotics (ISER)},
year    = {2023},
url={http://ais.informatik.uni-freiburg.de/publications/papers/nematollahi23iser.pdf},
address = {Chiang Mai, Thailand}
}

Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models
Iman Nematollahi^*, Erick Rosete-Beas^*, Adrian Röfer, Tim Welschehold, Abhinav Valada, Wolfram Burgard
arxiv
Pdf

Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models Iman Nematollahi^*, Erick Rosete-Beas^*, Adrian Röfer, Tim Welschehold, Abhinav Valada, Wolfram Burgard arxiv Pdf

BibTeX citation

                            
@inproceedings{nematollahi22icra,
author  = {Iman Nematollahi and Erick Rosete-Beas and Adrian Roefer and Tim Welschehold and Abhinav Valada and Wolfram Burgard},
title   = {Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models},
booktitle = {Proceedings of the IEEE International Conference on Robotics and Automation  (ICRA)},
year = {2022},
url={http://ais.informatik.uni-freiburg.de/publications/papers/nematollahi22icra.pdf},
address = {Philadelphia, USA}
}

Authors

Robot Skill Generalization via Keypoint IntegratedSoft Actor-Critic Gaussian Mixture Models