In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other situations in which it is better to just treat the probability as if it were the binary sample.

In regular Gibbs Sampling you would iterate:

$P(h=1) = \operatorname{sigmoid}(v*W)$

$P(v=1) = \operatorname{sigmoid}(W*h)$, sampling at each step.

But in the Contrastive Divergence described you iterate:

$P(h) = \operatorname{sigmoid}(P(v)*W)$

$P(v) = \operatorname{sigmoid}(W*P(h))$, and only sample after all iterations.

Is using the probability equivalent to sampling, or is it just an approximation? Why would using the probability be equivalent or a good approximation in some situations but not others? In what situations is using the probability valid in a Deep Boltzmann Machine?

  • $\begingroup$ Ok, I made the question more detailed. $\endgroup$ – Fantasy Apr 23 '18 at 20:23

From the paper you mentioned, section 3.3:

Assuming that the visible units are using real-valued probabilities instead of stochastic binary values, there are two sensible ways to collect the positive statistics for the connection between visible unit $i$ and hidden unit $j$:

$\langle p_ih_j\rangle_{data}$ or $\langle p_ip_j\rangle_{data}$

where $p_j$ is a probability and $h_j$ is a binary state that takes value 1 with probability $p_j$ . Using $h_j$ is closer to the mathematical model of an RBM, but using $p_j$ usually has less sampling noise which allows slightly faster learning.

Also see footnote 2 from the same page.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.