Abstract
Robotic manipulation of deformable linear objects (DLOs) has broad application prospects, e.g., manufacturing and medical surgery. To achieve such tasks, a critical challenge is the precise control of the DLOs' shapes, which requires an accurate dynamics model for deformation prediction. However, due to the infinite dimensionality of the DLOs and the complexity of their deformation mechanism, dynamics models are hard to theoretically calculate.
In this paper, for representing the DLO, we use multiple particles being uniformly distributed along the DLO. For learning the dynamics model, we adopt Graph Neural Network (GNN) to learn interaction effects between neighbouring particles, and use the attention mechanism to aggregate the effects of these interactions for the purpose of effect propagation along the DLO (called GA-Net). For manipulation, the Model Predictive Control (MPC) coupled with the learned dynamics model is used to calculate the optimal robot movements, which can also generalize to unseen DLOs.
Simulation and real-world experiments demonstrate that GA-Net shows better accuracy than existing methods, and the proposed control framework is effective for different DLOs. Specifically, for model prediction (150 steps), the prediction performance of GA-Net is 12.02% better than the strong baseline (IN-BiLSTM).
Overview of the proposed DLO shape control framework. (a) The training data is generated by random motion. (b) The DLO is represented by particles and the dynamics models predicts the DLO’s deformation based on GNN. (c) After obtaining the learned dynamics model, we apply the shooting method for trajectory optimization under the MPC framework.
Dynamics Model
Our goal is to learn a dynamics model for predicting the future state of a DLO, given the current state and the external effect.
Interaction Network (IN)
Proposed Method
IN: IN defines a flexible and efficient model for explicit reasoning of objects and their relations in a complex system. However, the main limitation of IN is that at every time step t, it only considers local information in the graph and cannot handle instantaneous propagation of effects.
Proposed Method: Proposed Method adopts the IN dynamics model for capturing the interaction between neighboring segments in a DLO and use an attention mechanism to propagate interaction effects along the DLO.
CONTROL USING LEARNED DYNAMICS
Trajectory Optimization under MPC Framework
The learned dynamics model is naturally differentiable. Given the model and a desired goal, we can perform forward simulation, optimizing the control inputs by minimizing a loss between simulated results and the goal.
The MPC is used to stabilize the trajectory by doing forward simulation at every time step as a way to compensate the simulation error
Real-world Experiments for Shape Control
A Kinova Gen3 robot manipulates one end of a DLO to different desired shapes, and the other end is fixed. In real-world experiments, the DLO is a 0.5-m USB cable and characterized by 8 features (red markers). An Intel Realsense D435 RGBD camera is utilized to obtain the positions of features of the DLO at 15Hz and 1280 x 720 resolution.
Real-world Setup
Shape Control Task 1
Shape Control Task 2
Shape Control Task 3
Shape Control Task 4
