1
$\begingroup$

In Basic LSTM cell of tensorflow there is an argument named forget_bias. From the documentation of tensorflow that I provided:

We add forget_bias (default: 1) to the biases of the forget gate in order to reduce the scale of forgetting in the beginning of the training.

It is not clear to me why LSTM forgets state in the beginning? So this argument wants to reduce that.

$\endgroup$
0
$\begingroup$

As a rule of thumb, typical good weight initialization schemes result in small starting values. The amount of cell state which is "forgotten" is typically in the form $1-\sigma(z)$ where $z$ is some function of input and hidden state. As pointed out before $z$ is likely initialized to some smallish value which we can approximate as $0$, meaning that the forget gate drops about half of the cell-state every step. By adding in a bias of 1, the amount forgotten is roughly only 0.27, which helps a bit.

$\endgroup$
  • $\begingroup$ i dont think that the forgot gate in usual act like you said, for example see here colah.github.io/posts/2015-08-Understanding-LSTMs. by the way, in usual gate, that is just σ(z), small start values can cause the forgetting. But in the other hand the input gate has a similar calculation. so why it does not need to add to one? $\endgroup$ – keramat Jan 20 at 15:47
  • $\begingroup$ @keramat the usual gate of $\sigma(z)$ is how much of the previous state is "remembered", so $1-\sigma(z)$ is how much is forgotten. $\endgroup$ – shimao Jan 20 at 15:53
  • $\begingroup$ ok,and why 1 does not need for input gate? $\endgroup$ – keramat Jan 20 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.