I put together some math to show what I'm talking about: https://drive.google.com/file/d/0BwbWRPtraa2zeDhaUWFUVl94ZUk/view?usp=sharing
TL;DR: the earlier the timestep, the more number of forget gate terms present in our derivative that multiply together. If one of these is equal or close to 0, then the whole gradient dies. How is this not an issue, if we train the forget gate [weights] simultaneously? Even if we set a really large forget bias, as the training progresses it'll correct this and make the forget gate at a timestep closer to what is optimal. Eg. if learn that f_10 should be 0, then the whole thing dies, even though we're still making contributions albeit small.
Am I missing something here?
