Please, I'm a newby with Restricted Boltzmann Machine, I'm a psychologist (not very good with math) and and I've some confusion about the use of Gaussian visible units.
Note I'm working on the original Hinton code, in particular:
- I'm using the Matlab code "rbm.m" (implementing a RBM with binary visible and binary hidden units);
- I'm using MNIST dataset as input;
- I'm reading Hinton paper "A Practical Guide to Training Restricted Boltzman Machine (2010)"
As I understand the following claims are corrected:
1) inputs (a grayscale 28x28 patch) are real values in [0,1], even if the the pixels distribution is mainly binary (almost all pixels are 0 or 1); I guess that "binary visible units" refers to input distribution.
2) hidden units are updated using the sigmoid function on the weighted sum of inputs, which computes the probability of the unit to be turned on or off; then hiddens are stocastichally turned on or off.
3) once hidden units are made binary, they are used (as input) to rebuild the visible units: the reconstructed visible units are updated using the sigmoid function on the weighted sum of hiddens. Hinton states (in the paper) that it is common to use this probability as the visible units instead of sampling binary values. I guess that "binary visible units" refers to this feature too.
4) finally, the reconstructed visible units (real values in [0,1]) are used to sample again the hidden units (same operation of point 2).
Now I come to my question. In the paper Hinton states that, if using natural images as input, binary visible units should be replaced with linear units with Gaussian noise.
Question 1: is Hinton speaking about the phase where the visible units are build from hidden units (point 3)? I mean, is he suggesting to use the simple weighted sum of hiddens (plus bias) in order to reconstruct the visible units?
Question 2: for "Independent Gaussian Noise", does it mean that natural images have a Gaussian distribution? Or should I add a Gaussian Noise to input?