Feedforward Nets, RNNs, and LSTMs Theory It is my understanding that feed-forward neural nets learn by backpropegating the error from comparing an outcome to the ground truth. How is this process not 'recurrent', as in recurrent nets, data is passed backwards through the model right? What is the difference between feed-forward nets and recurrent nets?
In addition, how do LSTM units play a role in recurrent nets? It is my understanding that LSTM cells 'remember' values over arbitrary time intervals, and the gates regulate the flow of data. But what exactly are these cells? Are they separate from the nodes of a recurrent net? And how do gates regulate the flow of data? Does this mean that some data skips passing over certain nodes?
 A: In FFn, a network responds with exactly the same output for a given input every time. This is not the case with RNN.
What is recurrent in RNNs is the fact that their internal state is used as a part of an input.
It allows to make RNN deal with variable-length inputs, which you can only emulate with FF. With RNN you can do like 'what is the likelihood of next letter being "a"' in a text and you can feed your RNN letter by letter and at each step (letter) a network will give you a response. Response will change over time. I.e, when you feed it with mathematical it may give different likelihood of a at step 2 (after m) than at step 7 (after 2nd m) even though inputs at these two steps are equal. This is possible, because RNN has a state being (recurrently) passed between calls to it.
Please note that in the example above, RRN can deal with words or text of any length.
By a 'call' i mean operation that yield output from a network, like a function call. In software engineering, recurrent function is one that calls itself. It is kind of implied that state is different between calls. Same case for RNNs but the 'state' thing is more pronounced here :)
LSTM is more sophisticated implementation of RNN. Internal state is more complex to deal with the vanishing gradients problem, but the idea is basically the same. Please refer to this great (and famous) piece for details: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
FFNs have a state too (weights) but this state depends on training data only and does not change after training is complete.
FFNs can't deal with variable length input directly, because they have to 'see' the whole input at the input layer. The shape of input layer depends on input size.
