What are exactly restricted boltzmann machines?

Question

I've been reading through some stuff related to Restricted Boltzmann Machines (RBM). By wikipedia for example I can see the graphical model is in general very similar to a standard Feed Forward Network (FFN), but on the other hand there's the training which is based on Gibbs Sampling.

What I don't get though is the difference, if there's any, between RBM and FFN, apart from the training process. The basic algorithm to train a ANN is gradient descent, but you can use in general other optimization methods, such as L-BFGS for example.

So if the difference is only the training method it doesn't seem to me they're different networks.

On the other hand some people when introduce RBM starts with probabilistic graphical models, which makes me think maybe an RBM is different from a FFN.

Can you clarify what exactly is an RBM then?

Answer 1

A feed forward neural network is just a function $f(x; \theta)$ of some inputs and its weights. There is a clear procedure for computing the values of each unit/layer in the network from the previous layer. You can ascribe your own interpretation for what the outputs mean (a probability value, an RL policy, a point estimate, etc) and train the model accordingly.

On the other hand, an RBM models $P(x; \theta)$. It doesn't have any outputs and can't do anything other than tell you the probability of the data given the parameters. You can't compute the value of any unobserved variables in the RBM, because they're random variables.