In FFn, a network responds with exactly the same output for a given input every time. This is not the case with RNN.
What is recurrent in RNNs is the fact that their internal state is used as a part of an input.
It allows to make RNN deal with variable-length inputs, which you can only emulate with FF. With RNN you can do like 'what is the likelihood of next letter being "a"' in a text and you can feed your RNN letter by letter and at each step (letter) a network will give you a response. Response will change over time. I.e, when you feed it with
mathematical it may give different likelihood of
a at step 2 (after
m) than at step 7 (after 2nd m) even though inputs at these two steps are equal. This is possible, because RNN has a state being (recurrently) passed between calls to it.
Please note that in the example above, RRN can deal with words or text of any length.
By a 'call' i mean operation that yield output from a network, like a function call. In software engineering, recurrent function is one that calls itself. It is kind of implied that state is different between calls. Same case for RNNs but the 'state' thing is more pronounced here :)
LSTM is more sophisticated implementation of RNN. Internal state is more complex to deal with the vanishing gradients problem, but the idea is basically the same. Please refer to this great (and famous) piece for details: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
FFNs have a state too (weights) but this state depends on training data only and does not change after training is complete.
FFNs can't deal with variable length input directly, because they have to 'see' the whole input at the input layer. The shape of input layer depends on input size.