3
$\begingroup$

I am reading Deep Learning by Goodfellow and he seems to imply that the latter structure for a RNN will lead to information loss over the first structure. In both of the pictures, we have a RNN. x is the exogenous variable and h is the hidden layer. o is the prediction. As you can see, in the second structure, we feed our prediction (output of the output layer) into the next time step, as opposed to the output of the hidden layer.

What is the intuition behind the information loss that occurs with the second structure? Is it because at the second time step, the neural network has less input variables and this implies less information?

First Structure

enter image description here

Second Structure enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

In the first one, the hidden state, $h$ is transferred to the future. Hidden state contains all the past information, and is richer compared to what is being outputted. However, in the second one, only the output variable, $o$ is transferred to the next state/future. What is to be transferred is decided based on hidden-to-output connections. So, some of the past information will be lost.

$\endgroup$
1
  • $\begingroup$ Ok thanks, that's what I thought. $\endgroup$
    – confused
    Commented Jul 29, 2020 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.