Boltzmann machines are stochastic recurrent neural networks with a specific architecture, and they are interesting in several contexts. They are stochastic in the sense that the sampling procedure involves repeatedly computing the activation functions for the next layer (hidden from visible or visible from hidden), sampling from the result, and using the obtained sample (as opposite to directly using the activation functions) for the next iteration.

I do wonder though, why is the stochasticity useful here?

If we want to compute the feature vector corresponding to a given input, do we use a sample from the hidden neurons' activations, or do we use directly the activations?

Using the same architecture as the RBM, but with deterministic neurons, would seem to also fit more naturally with the probabilistic energy-based interpretation: the activations on the visible layer after a pass through the hidden layer would directly correspond to a "proper" conditional probability of getting that output given the input, as opposite to just being the conditional probability of that output given a sample obtained from the conditional probability over the hidden layer given the used input.

The stochasticity seems to also make the training kind of weird. If the neurons are taken to be deterministic wouldn't the whole architecture effectively correspond to a regular feedforward NN with a given number of hidden layers (and some relations between the weights and biases of the layers), which could be trained using standard backpropagation?

  • $\begingroup$ You're asking how RBM is different from logistic (sigmoid) activated neural net? $\endgroup$
    – Aksakal
    Nov 21 '17 at 16:15
  • $\begingroup$ @Aksakal I guess that is part of the question yes. How is it different form a sigmoid activated neural net with same kind of topology, and more generally where/why does the stochasticity help. $\endgroup$
    – glS
    Nov 21 '17 at 16:18
  • $\begingroup$ So, you're not asking how stochasticity is used to find optimal weights? I mean annealing type of optimization is not what you're asking about $\endgroup$
    – Aksakal
    Nov 21 '17 at 16:21
  • $\begingroup$ @Aksakal I don't know, I'm not sure what you mean by that. If that is the reason why RBMs are chosen to be stochastic nets, then yes that would be a possible answer I think. It may be a very basic question, as I don't think I grasp the concept behind RBMs very well atm $\endgroup$
    – glS
    Nov 21 '17 at 16:22

Stochastic component in RBM (such as sampling from Boltzmann distribution) is used to find the optimal parameters of the RBM.

RBM itself is used to extract higher level features. You pre-train an RBM layer, then plug into the deep network. You train RBM in an unsupervised manner, somewhat similar to PCA or autoencoders.

For instance, AI people are used to think of PCA as a dimensionality reduction tool, which it is. However, it can be thought of a feature extractor too. Consider, interest rates on the market. You can borrow for 1 year or 10 years. If you go the market there would be quotes for borrowing to anything from overnight to 100 years, and each will have its own interest rate. It turns out that if you run these rates through PCA and look at first three components then something like this comes out: enter image description here So, the unsupervised (dumb) PCA was able to extract three features that are interpreted as level, slope and the curvature of the interest rate yield curve. These features are widely used in finance.

So, RBM serves a similar purpose in the neural net. It's pre-trained separately to extract some features from the input, then these features are passed along to the next layer in the network.

  • $\begingroup$ So am I understanding correctly that the stochasticity is only used at training time? But my question remains: why use a stochastic activation at all, instead of a more regular sigmoid activation, to extract said features? $\endgroup$
    – glS
    Nov 21 '17 at 17:02
  • $\begingroup$ It's the technical difficulty of calculating the gradient of energy for optimization routine. It turns out that the practical way is through some kind of Monte Carlo. See the last equation in the first section here. That gradient is impossible to deal with analytically $\endgroup$
    – Aksakal
    Nov 21 '17 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.