Accelerating Neural ODEs Using Model Order Reduction
Authors:
Mikko Lehtimäki,
Lassi Paunonen,
Marja-Leena Linne
Abstract:
Embedding nonlinear dynamical systems into artificial neural networks is a powerful new formalism for machine learning. By parameterizing ordinary differential equations (ODEs) as neural network layers, these Neural ODEs are memory-efficient to train, process time-series naturally and incorporate knowledge of physical systems into deep learning models. However, the practical applications of Neural…
▽ More
Embedding nonlinear dynamical systems into artificial neural networks is a powerful new formalism for machine learning. By parameterizing ordinary differential equations (ODEs) as neural network layers, these Neural ODEs are memory-efficient to train, process time-series naturally and incorporate knowledge of physical systems into deep learning models. However, the practical applications of Neural ODEs are limited due to long inference times, because the outputs of the embedded ODE layers are computed numerically with differential equation solvers that can be computationally demanding. Here we show that mathematical model order reduction methods can be used for compressing and accelerating Neural ODEs by accurately simulating the continuous nonlinear dynamics in low-dimensional subspaces. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and interpolation operations as layers of the neural network. We validate our approach by comparing it to neuron pruning and SVD-based weight truncation methods from the literature in image and time-series classification tasks. The methods are evaluated by acceleration versus accuracy when adjusting the level of compression. On this spectrum, we achieve a favourable balance over existing methods by using model order reduction when compressing a convolutional Neural ODE. In compressing a recurrent Neural ODE, SVD-based weight truncation yields good performance. Based on our results, our integration of model order reduction with Neural ODEs can facilitate efficient, dynamical system-driven deep learning in resource-constrained applications.
△ Less
Submitted 2 March, 2022; v1 submitted 28 May, 2021;
originally announced May 2021.
Decentralized Observer Design for Virtual Decomposition Control
Authors:
Jukka-Pekka Humaloja,
Janne Koivumäki,
Lassi Paunonen,
Jouni Mattila
Abstract:
In this paper, we incorporate velocity observer design into the virtual decomposition control (VDC) strategy of an $n$-DoF open chain robotic manipulator. Descending from the VDC strategy, the proposed design is based on decomposing the $n$-DoF manipulator into subsystems, i.e., rigid links and joints, for which the decentralized controller-observer implementation can be done locally. Similar to V…
▽ More
In this paper, we incorporate velocity observer design into the virtual decomposition control (VDC) strategy of an $n$-DoF open chain robotic manipulator. Descending from the VDC strategy, the proposed design is based on decomposing the $n$-DoF manipulator into subsystems, i.e., rigid links and joints, for which the decentralized controller-observer implementation can be done locally. Similar to VDC, the combined controller-observer design is passivity-based, and we show that it achieves semiglobal exponential convergence of the tracking error. The convergence analysis is carried out using Lyapunov functions based on the observer and controller error dynamics. The proposed design is demonstrated in a simulation study of a 2-DoF open chain robotic manipulator in the vertical plane.
△ Less
Submitted 27 May, 2021; v1 submitted 4 February, 2020;
originally announced February 2020.
