# forget_bias interpretation in tensorflow

In Basic LSTM cell of tensorflow there is an argument named forget_bias. From the documentation of tensorflow that I provided:

We add forget_bias (default: 1) to the biases of the forget gate in order to reduce the scale of forgetting in the beginning of the training.

It is not clear to me why LSTM forgets state in the beginning? So this argument wants to reduce that.

As a rule of thumb, typical good weight initialization schemes result in small starting values. The amount of cell state which is "forgotten" is typically in the form $$1-\sigma(z)$$ where $$z$$ is some function of input and hidden state. As pointed out before $$z$$ is likely initialized to some smallish value which we can approximate as $$0$$, meaning that the forget gate drops about half of the cell-state every step. By adding in a bias of 1, the amount forgotten is roughly only 0.27, which helps a bit.
• @keramat the usual gate of $\sigma(z)$ is how much of the previous state is "remembered", so $1-\sigma(z)$ is how much is forgotten. – shimao Jan 20 at 15:53