Roadmap to Boltzmann Machines

Question

In using the word roadmap I am referring to a series of book one should read in order to get acquainted with a concept or a field of study (following the usage of the word in MathOverflow). I am studying to become a probabilist and a few days ago I found and read part of this book on neuron models, which I found extremely fascinating. I talked about it with a professor of mine and he suggested me to read something about Boltzmann Machines (and Markov Random Fields, but that's another story). I know nothing about neural computation and this is why I am asking for a roadmap to be able to understand what it is about and, finally, what Boltzmann Machines are and why he suggested me that I should read about them.

I found this book by Johann Hertz through an answer given to a question similar to mine on Quora, but it is quite old and I am not all too sure about it.

Do you have any suggestions about what to read, be they publications or books?

Answer 1

Reading up books on neuron models is a good start, but it may not be the best way to start your journey to be an expert in probability and statistics. Some of the concepts in that area may leave you challenged. A better approach is to pick up something simpler and work your way up. Here is a collection of books from the relatively simple to more advanced ones that deal with Probability and machine learning. Hopefully you will find them interesting. There is also online course material that covers a lot, but you still need some background learning. e.g. http://deeplearning.net/tutorial/

Answer 2

My suggestion is about meeting point of statistical mechanics and machine learning. Others may suggest meeting points of machine learning and other disciplines.

Boltzmann Machines are stochastic graphical models inspired from statistical mechanics. Therefore, you can encounter with "energy" term while working with probabilistic graphical models. It is a scalar term that indicates the configuration of graphical models, and it is a metaphor from physics.

I think evolution map of Boltzmann Machines starts from Ising Model [1]. It is a mathematical model for electron spins in atoms, and the definition of energy term is used for descendant graphical models (including Boltzmann Machines).

The second stop is Hopfield Networks [2]. They are deterministic model since state transitions of nodes depend on energy difference between two states. The model memorizes input data by using weights within nodes (all of them are visible, no hidden nodes are introduced for this type of model).

The last stop is Boltzmann Machines (BM) [3]. They are stochastic counterpart of Hopfield Network. Energy difference between two states determines the probability of state transition. Hidden nodes are introduced in BM so that more representation power is available.

[1] https://page.mi.fu-berlin.de/rojas/neural/chapter/K13.pdf (I think that it is excellent book to understand intuition behind graphical models.)

[2] http://www.its.caltech.edu/~bi250c/papers/Hopfield-1982.pdf (Original paper by John Hopfield)

[3] https://www.enterrasolutions.com/media/docs/2013/08/cogscibm.pdf (Original paper from Ackley and Hinton, the original paper describes all formulas of Boltzmann Machines, you should check appendix part of the paper.)