Showing 1–2 of 2 results for author: Defourneau, T

Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings

Authors: Martin Gonzalez, Thibault Defourneau, Hatem Hajri, Mihaly Petreczky

Abstract: In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-outp… ▽ More In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-output map to be realizable by an ODE-LSTM and sufficient conditions for minimality of such systems. These results represent the first steps towards realization theory of recurrent neural ODE architectures, which is is expected be useful for model reduction and learning algorithm analysis of recurrent neural ODEs. △ Less

Submitted 1 August, 2022; v1 submitted 24 May, 2022; originally announced May 2022.

Comments: 10 pages. Corrected typos and added references

Journal ref: Systems & Control Letters 173 (2023)

Realization theory of recurrent neural networks and rational systems

Authors: Thibault Defourneau, Mihaly Petreczky

Abstract: In this paper, we show that, under mild assumptions, input-output behavior of a continous-time recurrent neural network (RNN) can be represented by a rational or polynomial nonlinear system. The assumptions concern the activation function of RNNs, and it is satisfied by many classical activation functions such as the hyperbolic tangent. We also present an algorithm for constructing the polynomial… ▽ More In this paper, we show that, under mild assumptions, input-output behavior of a continous-time recurrent neural network (RNN) can be represented by a rational or polynomial nonlinear system. The assumptions concern the activation function of RNNs, and it is satisfied by many classical activation functions such as the hyperbolic tangent. We also present an algorithm for constructing the polynomial and rational system. This embedding of RNNs into rational systems can be useful for stability, identifiability, and realization theory for RNNs, as these problems have been studied for polynomial/rational systems. In particular, we use this embedding for deriving necessary conditions for realizability of an input-output map by RNN, and for deriving sufficient conditions for minimality of an RNN. △ Less

Submitted 18 March, 2019; v1 submitted 13 March, 2019; originally announced March 2019.