Why do we fit Recurrent Neural Networks with backprop instead of message passing/expectation propagation?--as with hidden markov models

Question

The form of a Recurrent Neural Network (RNN) seems to resemble that of a hidden markov model. With a hidden markov model we have transitions between discrete states, as well as an emission variable that charts the observed measurement--from which we glean the hidden state. Hidden Markov models are usually fit using a forward/backward algorithm that is a specific instance of message passing based inference, as we would do in a graphical model. I guess if the model's hidden states are continuous, then we would use expectation propagation instead.

The point is, given that RNNs and hidden markov models look so similar, why can we use backpropagation through time for RNNs while we use message passing for hidden markov models. I was trying to understand the mathematical intuition behind this point.

Thanks.

Answer 1

The two models have distinct levels of uncertainty. Once you have the parameters and inputs of an RNN, all hidden states of the RNN are uniquely determined. For an HMM, even if you know the parameters and emissions, you don't know the hidden states. The forward-backward algorithm is needed to integrate out the uncertainty over the hidden states of the HMM. You can see the difference in the likelihood function for an HMM compared to an RNN: the HMM likelihood has a sum over hidden states, while the RNN does not. When you take the gradient of the HMM likelihood, this sum remains, and you get the forward-backward algorithm.

If you were doing Bayesian inference in an RNN, then the hidden states would be uncertain (since the parameters are uncertain) and there would be a reason to do message passing/expectation propagation.