Is there a correspondence between the hidden states of RNNs (LSTMs and GRUs) and the n-gram models?.

Karpathy et al gave an empirical evidence that the hidden state of LSTM does keeps track of structure in the sequence. Also these models are unable to capture features beyond a certain number of time steps (due to vanishing gradient), which is something similar to what an n-gram model does (keeps track of previous few timesteps only).

  • $\begingroup$ Markov models (hidden or not) are generative, so the whole setup is different (RNNs are discriminative). This is a bit like the discussion of neural ("predictive") word embeddings vs. the traditional distributional ("count-based") models. And overall, to me at least, it is not entirely clear what kind of insights you are hoping to get from an answer to your question? $\endgroup$ – fnl Jul 5 '17 at 8:25
  • $\begingroup$ But RNNs can be used to generate sequences and Markov models have been used for text classification. The insights which I was seeking is there a mathematical relation between the two. $\endgroup$ – euler16 Jul 5 '17 at 11:22
  • $\begingroup$ Don't get me wrong, from a standpoint of mathematical/scientific curiosity, the question is perfectly valid, I fully agree to that. What I fail to see is a direct, practical impact of working out the answer to it (much like the mentioned discussion a couple years back about word representations didn't make us any wiser, IMO). $\endgroup$ – fnl Jul 5 '17 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.