In Basic LSTM cell of
tensorflow there is an argument named
forget_bias. From the documentation of
tensorflow that I provided:
We add forget_bias (default: 1) to the biases of the forget gate in order to reduce the scale of forgetting in the beginning of the training.
It is not clear to me why LSTM forgets state in the beginning? So this argument wants to reduce that.