Looking for good references on Neural Networks (math for learning algorithms) I'm looking for a book that in detail covers the mathematical basis for different learning algorithms in order to better guide my intuition on what is difficult. My background is a Math Ph.D., but I'm fairly new to the NN area having mostly done some small-scale networks for simple problems as well as done some courses on Coursera/Udacity.
I'd like to understand in detail why it's difficult to do learning for certain types of NNs and what the mathematical basis of that is in order to better guide my intuition when designing networks. I know about Bishop's Neural Network for Pattern Recognition as well as Hassoun's Fundamentals of Artificial Neural Networks. However, these are fairly old, so they seems to not mention things like RNNs as there's been a whole lot of development in the field since.
Does anyone have any recommendations? If no books cover this, I wouldn't mind digging into papers covering the most important advances in the area. I'm trying to work myself up to the level of a grad student in the area. I started going to a seminar series organized by one of the top departments, which is almost next door, but there's a whole lot of fundamentals to figure out before I can better follow what's being discussed.
 A: I can recommend the book "Deep Learning" by Ian Goodfellow and Yoshua Bengio and Aaron Courville. It is available at
http://www.deeplearningbook.org/
It covers all aspects of ANNs and also the fairly new techniques. Here's the table of contents:
Part I: Applied Math and Machine Learning Basics


*

*Linear Algebra

*Probability and Information Theory

*Numerical Computation

*Machine Learning Basics


Part II: Modern Practical Deep Networks


*Deep Feedforward Networks

*Regularization for Deep Learning

*Optimization for Training Deep Models

*Convolutional Networks

*Sequence Modeling: Recurrent and Recursive Nets

*Practical Methodology

*Applications


Part III: Deep Learning Research


*Linear Factor Models

*Autoencoders

*Representation Learning

*Structured Probabilistic Models for Deep Learning

*Monte Carlo Methods

*Confronting the Partition Function

*Approximate Inference

*Deep Generative Models

A: I highly recommend Neural Network Design by Martin Hagan.
I believe this is exactly what you are looking for. I have a similar background as you and I worked through the book in a couple of weeks, working out some of the use cases described and it provided me with a very good understanding of the (various) inner workings of the main aspects of the (various) neural network designs.
A: A classic book which is maybe a bit old, but very good for what it contains (no deep learning, that is a newer topic), and it has a statistical point of view. So look at Brian Ripley's book.
