I'm completely new to neural networks but highly interested in understanding them. However it's not easy at all to get started.
Could anyone recommend a good book or any other kind of resource? Is there a must-read?
I'm thankful for any kind of tip.

  • 11
    There's also a course that Goeff Hinton is teaching via Coursera on Neural Networks that starts next week: coursera.org/course/neuralnets – Marc Shivers Sep 13 '12 at 20:28
  • That course looks interesting, but from the video and web page, I would guess it is not designed as an introduction. – Douglas Zare Sep 15 '12 at 13:39
  • @DouglasZare: I am finishing Hinton's course, and in some parts I was glad I first took the ML course by Andrew Ng. – Andrew Nov 28 '12 at 17:43
  • 1
    Yes, I took it, too. It was a great course, and no introduction. – Douglas Zare Nov 28 '12 at 19:55
  • Blogs and tutorials about neural networks for beginners learn-neural-networks.com – Фаиль Гафаров Oct 29 at 9:29
up vote 32 down vote accepted

Neural networks have been around for a while, and they've changed dramatically over the years. If you only poke around on the web, you might end up with the impression that "neural network" means multi-layer feedforward network trained with back-propagation. Or, you might come across any of the dozens of rarely used, bizarrely named models and conclude that neural networks are more of a zoo than a research project. Or that they're a novelty. Or...

I could go on. If you want a clear explanation, I'd listen to Geoffrey Hinton. He has been around forever and (therefore?) does a great job weaving all the disparate models he's worked on into one cohesive, intuitive (and sometimes theoretical) historical narrative. On his homepage, there are links to Google Tech Talks and Videolectures.net lectures he has done (on RBMs and Deep Learning, among others).

From the way I see it, here's a historical and pedagogical road map to understanding neural networks, from their inception to the state-of-the-art:

  • Perceptrons
    • Easy to understand
    • Severely limited
  • Multi-layer, trained by back-propogation
    • Many resources to learn these
    • Don't generally do as well as SVMs
  • Boltzmann machines
    • Interesting way of thinking about the stability of a recurrent network in terms of "energy"
    • Look at Hopfield networks if you want an easy to understand (but not very practical) example of recurrent networks with "energy".
    • Theoretically interesting, useless in practice (training about the same speed as continental drift)
  • Restricted Boltzmann Machines
    • Useful!
    • Build off of the theory of Boltzmann machines
    • Some good introductions on the web
  • Deep Belief Networks
    • So far as I can tell, this is a class of multi-layer RBMs for doing semi-supervised learning.
    • Some resources
  • 1
    Thank you a lot for this overview and those many resources! – Claudio Albertin Sep 14 '12 at 13:41
  • 2
    No problemo. Good luck in your neural network endeavors. – Stumpy Joe Pete Sep 14 '12 at 15:57

I highly recommend watching these lectures and use this as reading material. These lectures are on machine learning in general by Andrew Ng talks in length about neural networks and does try hard to make it accessible for beginners.

These are, in my opinion, very good books.

  • R. Rojas: Neural Networks
  • C. M. Bishop: Neural Networks for Pattern recognition

The books have some similarities: They are both around 500 pages long, and they are fairly old, from 1995. Nevertheless, they remain very useful. Both books start from scratch, by explaining what neural networks are. They provide clear explanations, good examples and good graphs to aid understanding. They explain in great detail the issues of training neural networks, in their many shapes and forms, and what they can and cannot do. The two books supplement each other very nicely, for what one cannot figure out with one book, one tends to find in the other.

Rojas has a section, which I particularly like, about implementing back-propagation over many layers in matrix form. It also has a nice section about fuzzy logic, and one about complexity theory. But then Bishop has lots of other nice sections.

Rojas is, I would say, the most accessible. Bishop is more mathematical and perhaps more sophisticated. In both books, the maths is mostly linear algebra and calculus of functions of multiple variables (partial derivatives and so on). Without any knowledge of these subjects, you probably would not find either of these books very illuminating.

I would recommend reading Rojas first.

Both books, obviously, have a lot to say about algorithms, but neither says much about specific implementations in code.

To me, these books provide the background, which make an on-line course, (such as Hinton's, on Coursera) understandable. The books also cover much more ground, and in far greater detail, than can be done online.

I hope this helps, and am happy to answer any questions about the books.

  • 2
    Welcome to the site, @Old_Mortality. Can you say anything about those books? What is good about them? Are they appropriate for people w/ some level of mathematical &/or coding sophistication? Which would you recommend the OP read 1st? Etc. – gung Feb 11 '16 at 23:26
  • Thank you for the suggestion. I have edited my answer above. – Old_Mortality Feb 13 '16 at 7:49

As other people have pointed out, there are a lot of (good) resources online and I have personally done some of them:

  • Ng's Intro to ML class on Coursera
  • Hinton's Neural Networks class on Coursera
  • Ng's deep learning tutorial
  • reading the relevant chapters in the original Parallel Distributed Processing

I want to draw attention to the fact that these expositions mostly followed the classical treatment where layers (summation and non-linearity together) are the basic units. The more popular and more flexible treatment implemented in most libraries such as torch-nn and tensorflow, now uses computation graph with auto-differentiation to achieve high modularity. Conceptually it is simpler and more liberating. I would highly recommend the excellent Stanford CS231n open course for this treatment.

For a rigorous, learning-theoretic treatment, you may want to consult Neural Networks by Anthony and Bartlett.

If you want a treatment from a more statistical viewpoint, have a look at Brian Ripley's "Pattern Recognition and Neural Networks". This book isn't introductory and presupposes some statistical background.

http://www.stats.ox.ac.uk/~ripley/PRbook/

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.