I'm completely new to neural networks but highly interested in understanding them. However it's not easy at all to get started.
Could anyone recommend a good book or any other kind of resource? Is there a must-read?
I'm thankful for any kind of tip.
$\begingroup$
$\endgroup$
6
asked Sep 13, 2012 at 16:54
-
11$\begingroup$ There's also a course that Goeff Hinton is teaching via Coursera on Neural Networks that starts next week: coursera.org/course/neuralnets $\endgroup$– Marc ShiversSep 13, 2012 at 20:28
-
$\begingroup$ That course looks interesting, but from the video and web page, I would guess it is not designed as an introduction. $\endgroup$– Douglas ZareSep 15, 2012 at 13:39
-
$\begingroup$ @DouglasZare: I am finishing Hinton's course, and in some parts I was glad I first took the ML course by Andrew Ng. $\endgroup$– AndrewNov 28, 2012 at 17:43
-
1$\begingroup$ Yes, I took it, too. It was a great course, and no introduction. $\endgroup$– Douglas ZareNov 28, 2012 at 19:55
-
$\begingroup$ Blogs and tutorials about neural networks for beginners learn-neural-networks.com $\endgroup$– Фаиль ГафаровOct 29, 2018 at 9:29
|
Show 1 more comment
7 Answers
$\begingroup$
$\endgroup$
2
Neural networks have been around for a while, and they've changed dramatically over the years. If you only poke around on the web, you might end up with the impression that "neural network" means multi-layer feedforward network trained with back-propagation. Or, you might come across any of the dozens of rarely used, bizarrely named models and conclude that neural networks are more of a zoo than a research project. Or that they're a novelty. Or...
I could go on. If you want a clear explanation, I'd listen to Geoffrey Hinton. He has been around forever and (therefore?) does a great job weaving all the disparate models he's worked on into one cohesive, intuitive (and sometimes theoretical) historical narrative. On his homepage, there are links to Google Tech Talks and Videolectures.net lectures he has done (on RBMs and Deep Learning, among others).
From the way I see it, here's a historical and pedagogical road map to understanding neural networks, from their inception to the state-of-the-art:
- Perceptrons
- Easy to understand
- Severely limited
- Multi-layer, trained by back-propogation
- Many resources to learn these
- Don't generally do as well as SVMs
- Boltzmann machines
- Interesting way of thinking about the stability of a recurrent network in terms of "energy"
- Look at Hopfield networks if you want an easy to understand (but not very practical) example of recurrent networks with "energy".
- Theoretically interesting, useless in practice (training about the same speed as continental drift)
- Restricted Boltzmann Machines
- Useful!
- Build off of the theory of Boltzmann machines
- Some good introductions on the web
- Deep Belief Networks
- So far as I can tell, this is a class of multi-layer RBMs for doing semi-supervised learning.
- Some resources
answered Sep 13, 2012 at 20:29
-
1$\begingroup$ Thank you a lot for this overview and those many resources! $\endgroup$ Sep 14, 2012 at 13:41
-
2$\begingroup$ No problemo. Good luck in your neural network endeavors. $\endgroup$ Sep 14, 2012 at 15:57
$\begingroup$
$\endgroup$
1
I highly recommend watching these lectures and use this as reading material. These lectures are on machine learning in general by Andrew Ng talks in length about neural networks and does try hard to make it accessible for beginners.
-
1$\begingroup$ Could you indicate the titles? Links may go dead in the future... $\endgroup$ May 29, 2019 at 9:27
$\begingroup$
$\endgroup$
2
These are, in my opinion, very good books.
- R. Rojas: Neural Networks
- C. M. Bishop: Neural Networks for Pattern recognition
The books have some similarities: They are both around 500 pages long, and they are fairly old, from 1995. Nevertheless, they remain very useful. Both books start from scratch, by explaining what neural networks are. They provide clear explanations, good examples and good graphs to aid understanding. They explain in great detail the issues of training neural networks, in their many shapes and forms, and what they can and cannot do. The two books supplement each other very nicely, for what one cannot figure out with one book, one tends to find in the other.
Rojas has a section, which I particularly like, about implementing back-propagation over many layers in matrix form. It also has a nice section about fuzzy logic, and one about complexity theory. But then Bishop has lots of other nice sections.
Rojas is, I would say, the most accessible. Bishop is more mathematical and perhaps more sophisticated. In both books, the maths is mostly linear algebra and calculus of functions of multiple variables (partial derivatives and so on). Without any knowledge of these subjects, you probably would not find either of these books very illuminating.
I would recommend reading Rojas first.
Both books, obviously, have a lot to say about algorithms, but neither says much about specific implementations in code.
To me, these books provide the background, which make an on-line course, (such as Hinton's, on Coursera) understandable. The books also cover much more ground, and in far greater detail, than can be done online.
I hope this helps, and am happy to answer any questions about the books.
-
3$\begingroup$ Welcome to the site, @Old_Mortality. Can you say anything about those books? What is good about them? Are they appropriate for people w/ some level of mathematical &/or coding sophistication? Which would you recommend the OP read 1st? Etc. $\endgroup$ Feb 11, 2016 at 23:26
-
1$\begingroup$ Thank you for the suggestion. I have edited my answer above. $\endgroup$ Feb 13, 2016 at 7:49
$\begingroup$
$\endgroup$
0
As other people have pointed out, there are a lot of (good) resources online and I have personally done some of them:
- Ng's Intro to ML class on Coursera
- Hinton's Neural Networks class on Coursera
- Ng's deep learning tutorial
- reading the relevant chapters in the original Parallel Distributed Processing
I want to draw attention to the fact that these expositions mostly followed the classical treatment where layers (summation and non-linearity together) are the basic units. The more popular and more flexible treatment implemented in most libraries such as torch-nn and tensorflow, now uses computation graph with auto-differentiation to achieve high modularity. Conceptually it is simpler and more liberating. I would highly recommend the excellent Stanford CS231n open course for this treatment.
For a rigorous, learning-theoretic treatment, you may want to consult Neural Networks by Anthony and Bartlett.
$\begingroup$
$\endgroup$
If you want a treatment from a more statistical viewpoint, have a look at Brian Ripley's "Pattern Recognition and Neural Networks". This book isn't introductory and presupposes some statistical background.
answered Feb 13, 2016 at 10:54
$\begingroup$
$\endgroup$
I have created a web application that supports your learning process in the field of neural networks.
You can play around with the settings (architecture, activation functions, training settings) and observe how the settings affect the predictions. All datasets have preconfigured values that can be adopted. It is also possible to create your own datasets.
Instructions and explanations to the implemented elements:
$\begingroup$
$\endgroup$
I'll throw my hat into the ring.
- Read / listen to multiple explanations from different people.
- Master the Perceptron before you attempt to learn Multilayer Perceptrons (i.e neural networks)
- As you learn concepts, try to implement them in code, from scratch
- Keep a few toy datasets and problems in your pocket for testing your understanding and your code
- Attempt to explain your knowledge to other people (for example, by answering questions on Cross Validated)
In regards to 5, when I learned neural networks, I created a video lecture series about them.
