1
$\begingroup$

Please, I'm a newby with Restricted Boltzmann Machine, I'm a psychologist (not very good with math) and and I've some confusion about the use of Gaussian visible units.

Note I'm working on the original Hinton code, in particular:

  • I'm using the Matlab code "rbm.m" (implementing a RBM with binary visible and binary hidden units);
  • I'm using MNIST dataset as input;
  • I'm reading Hinton paper "A Practical Guide to Training Restricted Boltzman Machine (2010)"

As I understand the following claims are corrected:

1) inputs (a grayscale 28x28 patch) are real values in [0,1], even if the the pixels distribution is mainly binary (almost all pixels are 0 or 1); I guess that "binary visible units" refers to input distribution.

2) hidden units are updated using the sigmoid function on the weighted sum of inputs, which computes the probability of the unit to be turned on or off; then hiddens are stocastichally turned on or off.

3) once hidden units are made binary, they are used (as input) to rebuild the visible units: the reconstructed visible units are updated using the sigmoid function on the weighted sum of hiddens. Hinton states (in the paper) that it is common to use this probability as the visible units instead of sampling binary values. I guess that "binary visible units" refers to this feature too.

4) finally, the reconstructed visible units (real values in [0,1]) are used to sample again the hidden units (same operation of point 2).

Now I come to my question. In the paper Hinton states that, if using natural images as input, binary visible units should be replaced with linear units with Gaussian noise.

Question 1: is Hinton speaking about the phase where the visible units are build from hidden units (point 3)? I mean, is he suggesting to use the simple weighted sum of hiddens (plus bias) in order to reconstruct the visible units?

Question 2: for "Independent Gaussian Noise", does it mean that natural images have a Gaussian distribution? Or should I add a Gaussian Noise to input?

$\endgroup$
1
$\begingroup$

I remember struggling with a similar thing. Did you look at the matlab implementation of the Hinton's Gaussian RBM? I guess that's for Gaussian hidden units and binary visible units, so it's the other way but still relevant.

In section 13.2 of his training paper he has a different formula for the energy, where the visible units have mean $a_i$ and variance $\sigma_i^2$. That's what he means by linear and independent noise. Each visible unit has those two parameters associated with it. Other than that, you are right in saying that you "use the simple weighted sum of hiddens (plus bias)".

Furthermore, he goes on and says that in practice one should normalize the data to have zero mean and unit variance. See this answer on how to achieve that. So in reality/practice you choose 1 as unit variance for the energy formula and perform the normalization step on your data.

Patric

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.