3
$\begingroup$

It is my understanding that feed-forward neural nets learn by backpropegating the error from comparing an outcome to the ground truth. How is this process not 'recurrent', as in recurrent nets, data is passed backwards through the model right? What is the difference between feed-forward nets and recurrent nets?

In addition, how do LSTM units play a role in recurrent nets? It is my understanding that LSTM cells 'remember' values over arbitrary time intervals, and the gates regulate the flow of data. But what exactly are these cells? Are they separate from the nodes of a recurrent net? And how do gates regulate the flow of data? Does this mean that some data skips passing over certain nodes?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

In FFn, a network responds with exactly the same output for a given input every time. This is not the case with RNN.

What is recurrent in RNNs is the fact that their internal state is used as a part of an input.

It allows to make RNN deal with variable-length inputs, which you can only emulate with FF. With RNN you can do like 'what is the likelihood of next letter being "a"' in a text and you can feed your RNN letter by letter and at each step (letter) a network will give you a response. Response will change over time. I.e, when you feed it with mathematical it may give different likelihood of a at step 2 (after m) than at step 7 (after 2nd m) even though inputs at these two steps are equal. This is possible, because RNN has a state being (recurrently) passed between calls to it.

Please note that in the example above, RRN can deal with words or text of any length.

By a 'call' i mean operation that yield output from a network, like a function call. In software engineering, recurrent function is one that calls itself. It is kind of implied that state is different between calls. Same case for RNNs but the 'state' thing is more pronounced here :)

LSTM is more sophisticated implementation of RNN. Internal state is more complex to deal with the vanishing gradients problem, but the idea is basically the same. Please refer to this great (and famous) piece for details: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

FFNs have a state too (weights) but this state depends on training data only and does not change after training is complete.

FFNs can't deal with variable length input directly, because they have to 'see' the whole input at the input layer. The shape of input layer depends on input size.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Couldn't you say that a FFN 'remembers' previous inputs since it's weights get updated once those inputs are passed in? So my thought is train a FFN on inputs A (the weights now being updated to reflect input A), then train that same FFN on inputs B. Wouldn't that output of the FFN have used information from input A's hidden layer (the updated weights) and inputs B, similar to how an RNN would? $\endgroup$
    – io_error
    Sep 11, 2018 at 7:31
  • $\begingroup$ Also, why can't FFNs deal with variable-length inputs? And now that I think about it, what would be an example of a variable-length input? Can an FFN not deal with it because the number of its input layers are fixed from the first pass of an input? $\endgroup$
    – io_error
    Sep 11, 2018 at 7:34
  • $\begingroup$ Please see my answer updates. Example of variable-length input is a word. This can be of any length. In FFN you have to fix the input layer at a certain size, say 'word with 10 letters max'. There is no such limitation in RNN, you can feed it letter by letter. You can't change input size in FFN, because it would create new weights that were not present during training. $\endgroup$
    – Marcin
    Sep 11, 2018 at 8:13
  • $\begingroup$ For the example of sentences as variable length inputs to an RNN, can the RNN handle these because they are fed in letter by letter? Is the size of the RNN's input layer also fixed? And would that mean that the RNN has only one input node (for a single letter)? $\endgroup$
    – io_error
    Sep 11, 2018 at 17:40
  • $\begingroup$ Letter by letter - yes. Input layers size is fixed in RNN, but it can be of any size, not necessarily a single number. $\endgroup$
    – Marcin
    Sep 11, 2018 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.