Learning Graph Dynamics with Interaction Effects Propagation for Deformable Linear Objects Shape Control

Robotic manipulation of deformable linear objects (DLOs) has broad application prospects, e.g., manufacturing and medical surgery. To achieve such tasks, a critical challenge is the precise control of the DLOs' shapes, which requires an accurate dynamics model for deformation prediction. However, due to the infinite dimensionality of the DLOs and the complexity of their deformation mechanism, dynamics models are hard to theoretically calculate.

In this paper, for representing the DLO, we use multiple particles being uniformly distributed along the DLO. For learning the dynamics model, we adopt Graph Neural Network (GNN) to learn local interaction effects between neighboring particles, and use the attention mechanism to aggregate the effects of these interactions for the purpose of effect propagation along the DLO (called GA-Net). For manipulation, the Model Predictive Control (MPC) coupled with the learned dynamics model is used to calculate the optimal robot movements, which can also generalize to unseen DLOs.

Simulation and real-world experiments demonstrate that GA-Net shows better accuracy than existing methods, and the proposed control framework is effective for different DLOs. Specifically, for model prediction (150 steps), the prediction performance of GA-Net is 12.02% better than the strong baseline (IN-BiLSTM).

Overview of the proposed DLO shape control framework. (a) The training data is generated by random motion. (b) The DLO is represented by particles, and the dynamics model predicts the DLO’s deformation. (c) After obtaining the learned dynamics model, we apply the shooting method for trajectory optimization under the MPC-based framework. (d) The proposed framework is evaluated in real-world experiments.

Our goal is to learn a dynamics model for predicting the future state of a DLO, given the current state and the external effect.

We adopt Graph Nueral Network (GNN) to learn local interaction effects between neighboring particles, and use the attention mechanism to propagate local interaction effects along the DLO.

For manipulation, we use the shooting method for trajectory optimization, and use Model Predictive Control (MPC) to stabilize the trajectory by optimizing actions and executing the first action at every time step as a way to compensate the prediction error.

A Kinova Gen3 robot manipulates one end of a DLO to different desired shapes, and the other end is fixed. In real-world experiments, the DLO is a 0.5-m USB cable and characterized by 8 features (red markers). An Intel Realsense D435 RGBD camera is utilized to obtain the positions of features of the DLO at 15 Hz and 1280 x 720 resolution.

The time duration of manipulation is 11 seconds for 10-horizon tasks (inference by Nvidia RTX3090).

Shape Control Task 1

Shape Control Task 2

Shape Control Task 3

Shape Control Task 4

Real-world Experiments for Generalization (inference by CPU)

We evaluate the generalization ability of the proposed shape control framework by shape control of different types of DLOs. The dynamics model is trained on USB Cable 1. For USB Cable 2 and Shoelace, they are untrained.