# Rigorous Logic of LSTM

I have read several introductory texts about LSTM online, but none of them give a rigorously mathematical explanation of what the model actually does. For instance, why does the forget gate forget any more than any of the other gates in the LSTM unit?

Which papers and/or books should I read if I want to get a deeper understanding of LSTM?

Learning to store information over extended time intervals via recurrent backpropagation takes a very long time, mostly due to insufficient, decaying error back ow. We brifely review Hochreiter's 1991 analysis of this problem, then address it by introducing a novel, efficient, gradient-based method called "Long Short-Term Memory" (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete time steps by enforcing constant error ow through constant error carrousels" within special units. Multiplicative gate units learn to open and close access to the constant error low. LSTM is local in space and time; its computational complexity per time step and weight is $$O(1)$$. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation, Elman nets, and Neural Sequence Chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long time lag tasks that have never been solved by previous recurrent network algorithms.