My current understanding of Restricted Boltzmann Machines (RBMs) is as follows. Please correct me if I'm wrong, as misunderstanding RBMs may be the cause of my question.
An RBM is an energy-based model that is trained by minimizing its energy function. Its energy function is manually constructed/chosen such that the parameter values that result in the lowest energies produce a useful weight structure, assigning high weights to connections between visible and hidden neurons that are highly correlated in their on/off behaviour.
Why would this result in feature extraction? Why is it most energetically favourable for each hidden neuron to become associated with a different class/pattern of data, e.g. pen strokes and each type of number in MNIST (below), rather than many hidden neurons responding to data of the same class? Or is there some principle besides minimization of energy that causes feature extraction? If so, what is it?
As appears in Larochelle et al.