# Differences between Multi-layer NN, Hopfield, Helmholtz and Boltzmann machines

There are several resources describing these types of networks, but none of them explicitly address the differences between them.

What are the differences between theses models?

## 1 Answer

Multilayer NN (MLP) and Hopfield networks are deterministic networks. Concretely, the first can be shown to estimate the conditional average on the target data. For details you may have a look at Bishop's book on neural networks.

The Hopfield is a deterministic recurrent neural network. Deterministic because once the initial state is given, its dynamics evolves following a Lyapunov function. See papers by Hopfield and Tank. It has been shown that it can solve combinatorial problems and learn time series.

Helmholtz and Boltzmann machines are stochastic networks, meaning that given an input, the state of the network does not converge to a unique state, but to an ensemble distribution. A probability distribution of the state of the neural network. They are the stochastic equivalent of the Hopfield network.

One can actually prove that in the limit of absolute zero, $T \rightarrow 0$, the Boltzmann machine reduces to the Hopfield model.

You may look at the early papers by Hinton on the topic to see the basic differences, and the new ones to understand how to make them work.

Also, the Boltzmann and Helmholtz machines are strongly related to Markov Random Fields and Conditional Random Fields, as explained here and here. This leads to development of algorithms for inference that can be applied to both kinds of models, as for example fractional belief propagation.