I'm looking for a book that in detail covers the mathematical basis for different learning algorithms in order to better guide my intuition on what is difficult. My background is a Math Ph.D., but I'm fairly new to the NN area having mostly done some small-scale networks for simple problems as well as done some courses on Coursera/Udacity.

I'd like to understand in detail why it's difficult to do learning for certain types of NNs and what the mathematical basis of that is in order to better guide my intuition when designing networks. I know about Bishop's Neural Network for Pattern Recognition as well as Hassoun's Fundamentals of Artificial Neural Networks. However, these are fairly old, so they seems to not mention things like RNNs as there's been a whole lot of development in the field since.

Does anyone have any recommendations? If no books cover this, I wouldn't mind digging into papers covering the most important advances in the area. I'm trying to work myself up to the level of a grad student in the area. I started going to a seminar series organized by one of the top departments, which is almost next door, but there's a whole lot of fundamentals to figure out before I can better follow what's being discussed.

  • $\begingroup$ can you give a bit more details of eg the seminar series you are following and what you understand so far (eg is it optimisation only or anything NN related etc). Most of the maths is high school stuff. I would say the intuitions are probably not in a book. you are better off looking in tutorials eg icml.cc/2016/?page_id=97 $\endgroup$ – seanv507 Nov 3 '16 at 9:57

I can recommend the book "Deep Learning" by Ian Goodfellow and Yoshua Bengio and Aaron Courville. It is available at


It covers all aspects of ANNs and also the fairly new techniques. Here's the table of contents:

Part I: Applied Math and Machine Learning Basics

  1. Linear Algebra
  2. Probability and Information Theory
  3. Numerical Computation
  4. Machine Learning Basics

Part II: Modern Practical Deep Networks

  1. Deep Feedforward Networks
  2. Regularization for Deep Learning
  3. Optimization for Training Deep Models
  4. Convolutional Networks
  5. Sequence Modeling: Recurrent and Recursive Nets
  6. Practical Methodology
  7. Applications

Part III: Deep Learning Research

  1. Linear Factor Models
  2. Autoencoders
  3. Representation Learning
  4. Structured Probabilistic Models for Deep Learning
  5. Monte Carlo Methods
  6. Confronting the Partition Function
  7. Approximate Inference
  8. Deep Generative Models

I highly recommend Neural Network Design by Martin Hagan.

I believe this is exactly what you are looking for. I have a similar background as you and I worked through the book in a couple of weeks, working out some of the use cases described and it provided me with a very good understanding of the (various) inner workings of the main aspects of the (various) neural network designs.


A classic book which is maybe a bit old, but very good for what it contains (no deep learning, that is a newer topic), and it has a statistical point of view. So look at Brian Ripley's book.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.