RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting

Motion forecasting is a task defined as the prediction of future states or trajectories based on historical observations. However, sometimes not all the information is relevant to the forecasting.

For example, a vehicle must pay attention only to traffic participants interacting or having a conflict with itself. In such cases, the hard attention mechanism, which only pays attention to the relevant information, may be helpful.

Industrial robots. Image credit: ISAPUT via Wikimedia, CC-BY-SA-4.0

A recent paper on arXiv.org proposes a reinforcement learning-based hard attention mechanism for motion forecasting. In order to further discriminate relative importance, soft attention is employed as a ranking mechanism. It is the first hybrid attention-based framework for motion forecasting.

The framework is validated for multi-agent trajectory and human skeleton motion forecasting and achieves state-of-the-art performance.

Motion forecasting plays a significant role in various domains (e.g., autonomous driving, human-robot interaction), which aims to predict future motion sequences given a set of historical observations. However, the observed elements may be of different levels of importance. Some information may be irrelevant or even distracting to the forecasting in certain situations. To address this issue, we propose a generic motion forecasting framework (named RAIN) with dynamic key information selection and ranking based on a hybrid attention mechanism. The general framework is instantiated to handle multi-agent trajectory prediction and human motion forecasting tasks, respectively. In the former task, the model learns to recognize the relations between agents with a graph representation and to determine their relative significance. In the latter task, the model learns to capture the temporal proximity and dependency in long-term human motions. We also propose an effective double-stage training pipeline with an alternating training strategy to optimize the parameters in different modules of the framework. We validate the framework on both synthetic simulations and motion forecasting benchmarks in different domains, demonstrating that our method not only achieves state-of-the-art forecasting performance, but also provides interpretable and reasonable hybrid attention weights.

Research paper: Li, J., Yang, F., Ma, H., Malla, S., Tomizuka, M., and Choi, C., “RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting”, 2021. Link to the article: https://arxiv.org/abs/2108.01316

Link to the project site: https://jiachenli94.github.io/publications/RAIN/


Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x