DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

The task in which a robot manipulates a 3D deformable object into the desired shape is known as shape servo. The robot has to estimate the state of the object and use it as a feedback signal.

Previous learning-based methods to solve this problem focus on 1D or 2D objects as rope or cloth. A recent paper proposes the first solution to this problem for 3D shape servoing.

Industrial robots. Image credit: Auledas via Wikimedia, CC-BY-SA-4.0

The authors create a deep neural network that takes point clouds of the deformable objects as the inputs and outputs feature vectors. They are later mapped to the desired end-effector’s position. After training, the robot computes the position of its gripper from the point clouds of the object’s current and goal shapes.

The researchers also look into the problem of choosing the best manipulation point. Experimental evaluation shows that the proposed approach deforms objects of a large number of shapes and outperforms previous methods.

In this paper, we propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. Controlling the shape of a 3D object requires an effective state representation that can capture the full 3D geometry of the object. Current methods work around this problem by defining a set of feature points on the object or only deforming the object in 2D image space, which does not truly address the 3D shape control problem. Instead, we explicitly use 3D point clouds as the state representation and apply Convolutional Neural Network on point clouds to learn the 3D features. These features are then mapped to the robot end-effector’s position using a fully-connected neural network. Once trained in an end-to-end fashion, DeformerNet directly maps the current point cloud of a deformable object, as well as a target point cloud shape, to the desired displacement in robot gripper position. In addition, we investigate the problem of predicting the manipulation point location given the initial and goal shape of the object.

Research paper: Thach, B., Kuntz, A., and Hermans, T., “DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation”, 2021. Link: https://arxiv.org/abs/2107.08067

Source