Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning

A self-driving car must be able to explore a new city such that it can learn to traverse from any starting location to any destination, a problem known as goal-conditioned reinforcement learning (GCRL).

Reinforcement Learning is used to improve navigation of autonomous vehicles. Image credit: Google

A recent paper proposes a novel approach to learn an agent that can tackle long-horizon GCRL tasks.

The researchers use successor features (SF), a representation that captures transition dynamics, to define a novel distance metric. The metric serves as a distance estimate and enables the computation of a goal-conditioned function without further learning.

A single self-supervised learning component that captures SF is used to build all the components of a graph-based planning framework. It enables knowledge sharing between each module and stabilizes the overall learning. It is shown that the proposed approach outperforms state-of-the-art navigation baselines, most notably when goals are furthest away.

Operating in the real-world often requires agents to learn about a complex environment and apply this understanding to achieve a breadth of goals. This problem, known as goal-conditioned reinforcement learning (GCRL), becomes especially challenging for long-horizon goals. Current methods have tackled this problem by augmenting goal-conditioned policies with graph-based planning algorithms. However, they struggle to scale to large, high-dimensional state spaces and assume access to exploration mechanisms for efficiently collecting training data. In this work, we introduce Successor Feature Landmarks (SFL), a framework for exploring large, high-dimensional environments so as to obtain a policy that is proficient for any goal. SFL leverages the ability of successor features (SF) to capture transition dynamics, using it to drive exploration by estimating state-novelty and to enable high-level planning by abstracting the state-space as a non-parametric landmark-based graph. We further exploit SF to directly compute a goal-conditioned policy for inter-landmark traversal, which we use to execute plans to “frontier” landmarks at the edge of the explored state space. We show in our experiments on MiniGrid and ViZDoom that SFL enables efficient exploration of large, high-dimensional state spaces and outperforms state-of-the-art baselines on long-horizon GCRL tasks.

Research paper: Hoang, C., Sohn, S., Choi, J., Carvalho, W., and Lee, H., “Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning”, 2021. Link: https://arxiv.org/abs/2111.09858

Source