3
$\begingroup$

I'm just starting to learn about Restricted Boltzmann Machines (RBMs). So far I understand that they try to predict/output the inputs that are fed to them. If the RBM is trained on input where each record is a vector, then the RBM should likewise learn to output a vector.

While reading about RBMs, I keep coming across references to an "energy function", which I don't really understand, but my guess is that it represents the formula of a trained RBM. While reading through Geoffrey Hinton's tutorial I came across the following:

enter image description here

What confuses me about this paragraph is that it seems to suggest that the energy function outputs a scalar not a vector, which is inconsistent with my understanding above. Does it not actually output a scalar? Does an energy function not actually represent the architecture of a trained RBM and the means of calculating an output/prediction? Also, the article goes on to state that the variables a and b correspond to biases. If they're actually biases, why are they being multiplied with the vectors v and h? Aren't biases always supposed to be added only?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The Restricted Boltzmann Machine is an Energy - based model. The energy function produces a scalar value which basically corresponds to the configuration of the model and it is an indicator of the probability of the model being in that configuration. If the model is configured to favor low energy, then configurations leading to low energy will have a higher probability. Learning the model means looking for configurations that modify the shape of the energy function to drive to low energy configurations. As far as the biases go you basically calculate the dot product between the biases and the corresponding units (visible or hidden) to calculate their contribution to the energy function.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So you're saying that an RBM doesn't output a vector representing the input but produces the input's probability instead? $\endgroup$
    – Ryan Zotti
    Commented Jun 1, 2016 at 13:59
  • 1
    $\begingroup$ The RBM is a generative stochastic neural network. The parameters are adjusted to learn a probability distribution over the set of inputs. The energy function is not the output of the RBM and it is a scalar. I recommend you to read an introductory tutorial on RBMs. Check this out: deeplearning4j.org/restrictedboltzmannmachine.html. It will answer many of your questions $\endgroup$
    – kostas
    Commented Jun 1, 2016 at 20:26
  • $\begingroup$ Thanks for the link. I've actually read that article several times before, and it is pretty helpful for other aspects of RBMs, but it still leaves me confused. Is the energy function used to calculate the magnitude of error on an output so that the weights can be adjusted accordingly? If so, what is the energy function output compared against? Is the energy function purely a theoretical construct that is not used during model training? $\endgroup$
    – Ryan Zotti
    Commented Jun 1, 2016 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.