This question is about the use of Rectified Linear Units as hidden units in Restricted Boltzmann Machines.

In Nair and Hinton's paper, using ReLUs as hidden units is proposed. In section 1, they discuss the use of Bernoulli and Gaussian units (nothing new here). They provide the energy of an RBM with Guassian hidden units as

$$E(\textbf{v, h}) = \sum \frac{(v_i-b_i)^2}{2\sigma_i^2} - \sum b_j h_j - \sum \frac{v_i}{\sigma_i} h_j w_{ij}.$$

Sampling from the hidden units is easy as it is easy to interpret the output of a sigmoid function as a probability.

In section 2 they introduce ReLUs as hidden units. There are two things that I don't understand

  1. How should the energy function change?
  2. How should max(0,x) be interpreted as a probability?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.