RNN vs Kalman filter : learning the underlying dynamics?

Question

Being recently interested in Kalman filters and Recurrent neural networks, it appears to me that the two are closely related, yet I can't find relevant enough litterature :

In a Kalman filter, the set of equations is : $$x_{k} = Ax_{k-1} + Bu_{k} + w_{k-1}$$ $$ z_k = Hx_k + v_k$$

with $x$ the state and $z$ the measurement.

In an Elman RNN (from here), the relation between the layers is: $$h_{k} = \sigma_h (Uh_{k-1} + Wx_{k} + b)$$ $$ y_k = \sigma_y (Vh_k + c)$$

with $x$ the input layer, $h$ the hidden layer and $y$ the output layer and $\sigma$ are the activation functions for the layers.

It's clear that the two set of equations are the same, modulo the activations. The analogy here seems to be the following. The output layer corresponds to the measured state, the hidden layer is the true state, driven by a process $x$ which is the input layer.

• First question : is the analogy viable ? And how can we interpret the activations ?

• Second question : in a Kalman filter the $A$ matrix is that of the underlying dynamics of the state $x$. Since training a RNN allows to learn the $W$ matrices, are RNN able to learn the dynamics of the underlying state ? Ie once my RNN is trained, can I look at the coefficients of my network to guess the dynamics behind my data ?

(I'm going to try to do the experiment on artificially generated data, to see if this works, and will update as soon as it's done)

EDIT : I wish I had access to this paper

Answer 1

Yes indeed they are related because both are used to predict $y_{n}$ and $s_{n}$ at time step n based on some current observation $x_{n}$ and state $s_{n-1}$ i.e. they both represent a function $F$ such that $$F(x_{n}, s_{n-1}) = (y_{n}, s_{n})$$ The advantage of the RNN over Kalman filter is that the RNN architecture can be arbitrarily complex (number of layers and neurons) and its parameters are learnt, whereas the algorithm (including its parameters) of Kalman filter is fixed.

Recurrent Neural Networks are more general than Kalman filter. One could actually train a RNN to simulate a Kalman filter.
Neural nets are kind of black box models and weights and activations are very often not interpretable (above all in the deeper layers).

In the end neural nets are only optimized to make the best predictions and not to have "interpretable" parameters.
Nowadays if you work on time series, have enough data and want the best accuracy, RNN is the preferred approach.

Answer 2

As you say, the difference is the activation functions.

The usual purpose of a Kalman filter is used to model an intrinsically linear process, where the observations are subject to additive noise. You can get away with using a Kalman filter if there are slow deviation from linearity, but not if the process is strongly non-linear.

By contrast, neural nets get extra power by using a non-linear activation function. (Without it, stacking extra layers would never give a non-linear model.) So they are suitable for modelling non-linear processes, at least within the convex cover of the training set.

The asymptotic behaviour of a neural net is decided by the activation function, not by the training data. That is why the Universal Approximation Theorem has the condition "... on compact support". This isn't very surpising - no finite training set contains evidence about asymptotic behaviour.

Answer 3

They replace the kalman gain calculations by a RNN on this paper if you're still interested.

Answer 4

I'll substitute linear Gaussian state space model for Kalman filter here.

Similarities

• they both model time series
• they both have a hidden/latent "state" or "layer" process for which there is no data
• the observed dependent sequence depends/conditions on the above process
• the hidden/latent state/layer can depend on independent predictors/inputs/covariates/exogenous variables, etc.
• they both have parameters that (usually) must be learned/estimated

Differences

• RNN have nonlinear activation functions while linear Gaussian state space models have linear state equations and observation equations
• Linear Gaussian state space models have additive noise terms while RNN's do not

Misconceptions

1. RNNs are a class of models and Kalman filters are an algorithm. This makes comparison between the two misleading.
2. Kalman filters assume the parameters of the time series model are known, but that does not mean the models they are used on--linear Gaussian state space models--cannot have their parameters estimated/learned--they usually are, and the Kalman filter's likelihood evaluations can be used in an optimization or sampling-based strategy.

I have little experience with RNNs, but I tend to think that the last item in the "Differences" category is the most important. For example, RNNs such as long short term memory (LSTM) models and gated recurrent units (GRUs) have "forget gates" that describe how the hidden layer "forgets" its past values deterministically. On the other hand, a state space model's "forgetting" is random.

In my very humble opinion, I think this probably explains why RNNs are used a lot for non-noisy data such as text data, while lgSSMs are used on data that is "more random" such as financial returns. To be completely truthful, though, again, I don't have much experience with RNNs, so I don't claim this with any certainty.

The linear/nonlinear distinction is not so important. State space models can have nonlinear state and observation/emission equations and indeed frequently do. When this happens, though, you can't use the Kalman filter anymore, but there are many other approximate filtering techniques.

Regarding the other answer supposing that RNNs are "arbitrarily complex"--that reminds me of a Tweet I read a while back: https://twitter.com/sirbayes/status/1537177495866327040?s=20&t=eJ8U-Az5Tn_P0vH1afKLAw I have a hard time engaging in this debate, myself, though.