Home > Robotics > Ditto: Building Digital Twins of Articulated Objects from Interaction

Ditto: Building Digital Twins of Articulated Objects from Interaction

As modern AI models become larger, lots of realistic training data are needed. A recent paper published on arXiv.org introduces Ditto (Digital twin of articulated objects), an implicit neural representation-based model that jointly predicts part-level geometry and kinematic articulation between the parts.

Augmented reality application example. Image credit: OyundariZorigtbaatar via Wikimedia, CC-BY-SA-4.0

The main challenge is to establish correspondences between partial point cloud observations of an articulated object before and after interaction with one of its parts. To achieve this aim, researchers propose to encode the point clouds into two sets of subsampled point features, fuse them with a self-attention layer and decode them into dense point features.

It was shown that Ditto achieves superior results compared with the baselines. Moreover, researchers provide real-world examples of instantiating digital twins in simulation for a virtual robot to interact and transfer the interaction back to the real world.

Digitizing physical objects into the virtual world has the potential to unlock new research and applications in embodied AI and mixed reality. This work focuses on recreating interactive digital twins of real-world articulated objects, which can be directly imported into virtual environments. We introduce Ditto to learn articulation model estimation and 3D geometry reconstruction of an articulated object through interactive perception. Given a pair of visual observations of an articulated object before and after interaction, Ditto reconstructs part-level geometry and estimates the articulation model of the object. We employ implicit neural representations for joint geometry and articulation modeling. Our experiments show that Ditto effectively builds digital twins of articulated objects in a category-agnostic way. We also apply Ditto to real-world objects and deploy the recreated digital twins in physical simulation. Code and additional results are available at this https URL

Research paper: Jiang, Z., Hsu, C.-C., and Zhu, Y., “Ditto: Building Digital Twins of Articulated Objects from Interaction”, 2022. Link: https://arxiv.org/abs/2202.08227


Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x