I'm looking for a book that in detail covers the mathematical basis for different learning algorithms in order to better guide my intuition on what is difficult. My background is a Math Ph.D., but I'm fairly new to the NN area having mostly done some small-scale networks for simple problems as well as done some courses on Coursera/Udacity.
I'd like to understand in detail why it's difficult to do learning for certain types of NNs and what the mathematical basis of that is in order to better guide my intuition when designing networks. I know about Bishop's Neural Network for Pattern Recognition as well as Hassoun's Fundamentals of Artificial Neural Networks. However, these are fairly old, so they seems to not mention things like RNNs as there's been a whole lot of development in the field since.
Does anyone have any recommendations? If no books cover this, I wouldn't mind digging into papers covering the most important advances in the area. I'm trying to work myself up to the level of a grad student in the area. I started going to a seminar series organized by one of the top departments, which is almost next door, but there's a whole lot of fundamentals to figure out before I can better follow what's being discussed.