FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

Cloth manipulation is a challenging task for robot manipulation as fabrics do not transform rigidly when manipulated.

A recent paper introduces FabricFlowNet, a goal-conditioned policy for bimanual cloth manipulation that uses optical flow to improve policy performance.

Image credit: Pxfuel, Free licence

An optical flow-type network is used to estimate the relationship between the current observation and a sub-goal. The method is learned with supervised learning, relying on random actions without any expert demonstrations. The learned policy can perform bimanual manipulation and switches easily between dual and single-arm actions, depending on what is most suitable for the desired goal.

Experiments on a dual-arm robot system and in simulation show that FabricFlowNet outperforms state-of-the-art model-based and model-free baselines. It also generalizes with no additional training to other cloth shapes and colors.

We address the problem of goal-directed cloth manipulation, a challenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation in video, can also provide an effective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and as an action representation to improve performance. FabricFlowNet also elegantly switches between bimanual and single-arm actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-art model-free and model-based cloth manipulation policies that take image input. We also present real-world experiments on a bimanual system, demonstrating effective sim-to-real transfer. Finally, we show that our method generalizes when trained on a single square cloth to other cloth shapes, such as T-shirts and rectangular cloths. Video and other supplementary materials are available at: this https URL.

Research paper: Weng, T., Bajracharya, S., Wang, Y., Agrawal, K., and Held, D., “FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy”, 2021. Link: https://arxiv.org/abs/2111.05623

Source