My current understanding of Restricted Boltzmann Machines (RBMs) is as follows. Please correct me if I'm wrong, as misunderstanding RBMs may be the cause of my question.

An RBM is an energy-based model that is trained by minimizing its energy function. Its energy function is manually constructed/chosen such that the parameter values that result in the lowest energies produce a useful weight structure, assigning high weights to connections between visible and hidden neurons that are highly correlated in their on/off behaviour.

Why would this result in feature extraction? Why is it most energetically favourable for each hidden neuron to become associated with a different class/pattern of data, e.g. pen strokes and each type of number in MNIST (below), rather than many hidden neurons responding to data of the same class? Or is there some principle besides minimization of energy that causes feature extraction? If so, what is it?

Filters learned with an RBM from MNIST

As appears in Larochelle et al.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.