The form of a Recurrent Neural Network (RNN) seems to resemble that of a hidden markov model. With a hidden markov model we have transitions between discrete states, as well as an emission variable that charts the observed measurement--from which we glean the hidden state. Hidden Markov models are usually fit using a forward/backward algorithm that is a specific instance of message passing based inference, as we would do in a graphical model. I guess if the model's hidden states are continuous, then we would use expectation propagation instead.

The point is, given that RNNs and hidden markov models look so similar, why can we use backpropagation through time for RNNs while we use message passing for hidden markov models. I was trying to understand the mathematical intuition behind this point.


  • $\begingroup$ Do this and this give you some elements ? $\endgroup$ – TheCG Jul 21 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.