I have read several introductory texts about LSTM online, but none of them give a rigorously mathematical explanation of what the model actually does. For instance, why does the forget gate forget any more than any of the other gates in the LSTM unit?
Which papers and/or books should I read if I want to get a deeper understanding of LSTM?
You could always start at the beginning! I think the explanation here makes sense: LSTMs are effective because the specialized units improve training with back-propagation.
Sepp Hochreiter, and Jurgen Schmidhuber "Long Short Term Memory" Neural Computation 9(8):1735
Learning to store information over extended time intervals via recurrent backpropagation takes a very long time, mostly due to insufficient, decaying error back ow. We brifely review Hochreiter's 1991 analysis of this problem, then address it by introducing a novel, efficient, gradient-based method called "Long Short-Term Memory" (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete time steps by enforcing constant error flow through constant error carrousels" within special units. Multiplicative gate units learn to open and close access to the constant error low. LSTM is local in space and time; its computational complexity per time step and weight is $O(1)$. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation, Elman nets, and Neural Sequence Chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long time lag tasks that have never been solved by previous recurrent network algorithms.
I've found several blog posts to be detailed and helpful, though maybe not as mathematically rigorous as you'd like.
The forget gate is the output of a sigmoid function that multiplies the incoming memory value. The sigmoid forces its value to be in the range [0, 1] so: it can preserve (1), erase (0), or decrease (intermediate value) the incoming memory. The other contributor to the outgoing memory is by addition rather than multiplication.
So the forget gate can preserve a (incoming) memory or forget it, and the other LSTM cell logic doesn't affect the (incoming) memory. Hence, its name.
