This question is about the use of Rectified Linear Units as hidden units in Restricted Boltzmann Machines.

In Nair and Hinton's paper, using ReLUs as hidden units is proposed. In section 1, they discuss the use of Bernoulli and Gaussian units (nothing new here). They provide the energy of an RBM with Guassian hidden units as

$$E(\textbf{v, h}) = \sum \frac{(v_i-b_i)^2}{2\sigma_i^2} - \sum b_j h_j - \sum \frac{v_i}{\sigma_i} h_j w_{ij}.$$

Sampling from the hidden units is easy as it is easy to interpret the output of a sigmoid function as a probability.

In section 2 they introduce ReLUs as hidden units. There are two things that I don't understand

  1. How should the energy function change?
  2. How should max(0,x) be interpreted as a probability?

1 Answer 1


I have to agree the exposition in this paper was quite sparse. I think answering these two questions requires a bit of review of previous work, but you can skip to the bottom if you just want the answers.

Recall the energy function of binary RBMs.

$$E(v,h) = v^TWh + v^Ta + h^Tb $$

In the binomial RBM work, they duplicate each visible unit a bunch of times, and correspondingly the bias $a_i$ of each visible unit and the rows $W_i$ corresponding to "connections" to the hidden units are also copied.

Now, let $W'$, $a'$, and $v'$ denote the appropriately duplicated values of $W$, $a$, and $v$. Critically, the energy function does not change, other than swapping out some symbols:

$$E(v',h) = v'^TW'h + v'^Ta' + h^Tb $$

From an implementation and computational standpoint, it's quite wasteful to do extra computation for all those units. You might ask if we can save any effort by representing $K$ duplicates of a binary unit using a single variable which just stores how many of those $K$ duplicates are set to 1 (and how many are set to 0). I'll use $v^*$ to denote the combined duplicates: in other words, $v^*$ is the sum of each group of $K$ duplicates in $v'$

In order to do so, we need to make sure we can still sample hidden units given visible units and vice versa, which is a prerequisite for running contrastive divergence.

Sampling hidden from visible is easy because $P(h=1|v') = \sigma(b+W'^Tv') = \sigma(b+Wv^*)$

Sampling visible from hidden is slightly tricker. We have: $P(v'=1|h) = \sigma(a'+W'h)$. Now note that $v^*|h \sim \text{Binomial}(K, \sigma(a+Wh))$. By grouping out duplicated units together, we have gone from sampling from a bernoulli distribution to sampling from a binomial one.

Note that the gradient of the log probability:

$$\frac{\partial \log p(v)}{\partial W} = E_\text{data}[vh^T] - E_\text{model}[vh^T]$$

remains unchanged with our binomial units.

Now suppose after duplicating each unit $K$ times, we added different fixed biases to the duplicated units: the first duplicate bias is offset by $-0.5$, the second by $-1.5$, the third $-2.5$, and so on. In fact, why stop at $K$ duplicates -- why not make infinite duplicates of each unit. This is not as unreasonable as it might first sound, because each successive duplicate has a greater negative bias offset, so it has a vanishingly small chance of ever being switched on.

Whereas in the binomial case it was nice that we were able to group duplicate units together, here is it actually crucial, since computing with infinite binary units is not exactly possible. Whereas in the binomial case the sum of the duplicated units followed a binomial distribution, here it is possible to prove that the sum of the duplicated units approximately follows a distribution


Take care to note that $\sigma$ here denotes the sigmoid function and not the standard deviation. So instead of sampling from a binomial distribution, we sample from this relu'd normal distribution, and everything else proceeds as previously described.

Now that we have an understanding of what exactly it means to have relu activations in an RBM, we can return to the questions at hand.

How should the energy function change?

We have seen above that it does not change at all!

How should max(0,x) be interpreted as a probability?

It shouldn't, and the relu RBM model never uses relu as a probability value of any sort. relu is only used to approximate the sampling of infinite binary units.

  • 1
    $\begingroup$ Hey @shimao, thanks for the great explanation, just a clarification but in this case, when we sample from the infinite binary units, how do we interpret the final value? Isn't this different from our input where we only had one visible unit per pixel (in the typical case of a image)? $\endgroup$
    – q.Then
    Apr 20, 2020 at 15:14
  • $\begingroup$ @q.Then sorry for the very late reply -- i believe in the relu rbm paper they used relu hidden units but gaussian visible units, so one gaussian unit == 1 pixel. however if you wanted to use relu visible units as well, then the sum of each set of infinite binary units would correspond to one pixel. $\endgroup$
    – shimao
    May 16, 2020 at 1:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.