I am trying to confirm whether or not I am understanding the process described in the title.

I am implementing a CRMB (with Real Valued Gaussian Visible units and Binary hidden units) as outlined in Lee et al (2009b), explained deeper in Lee et al (2009a).

I understand sampling at the hidden layer (Binary Units) to involve:

  • For each group of hidden units, for each hidden unit find the probability of it being "on", given the input $v$:

$$P(h_j^k = 1|v) = sigmoid \big((\tilde{W}^k *_v v)_j + b_k\big)$$

where $*_v$ is the "valid" convolution - valid convolution of the transpose of group $k$'s weight matrix with the input.

  • Now to get the final Sample, for each hidden unit, you sample once from a Binomial Distribution with std dev = 1 and mean = the hidden unit's probability of being "on" - if this sampled value < the probability of the unit being "on", set the value of the hidden unit = 1.

This creates the "Sampled" value of the hidden layers - which will then be used as the starting point in the chain of Gibbs sampling in the Contrastive Divergence algorithm.

Now sampling at the visible layer (Gaussian Real units)

The papers listed define the probability of each visible unit being "on", given a group of Hidden Unit binary vectors:

$$P(v_i|\mathbf{h}) = Normal \Big(\sum_k (W^k *_f h^k)_i + c, 1\Big)$$

where $*_f$ is the "full" convolution - the "full" convolution of group $k$'s weight matrix with the $k$th group of hidden unit values.

They do not elaborate on the function:


Does this mean to sample the value from a normal distribution with mean = $x$ and std dev = $y$?

Does this also mean then that the probability of a visible unit being on IS the sample of that visible unit?

I've added a Python function below to hopefully explain how I believe this sampling is defined to work in a less ambiguous way:

def sample_v_given_h(self, hidden_groups):
    pre_sampled = np.zeros(self.visible_layer_shape)
    for group in range(1, self.numBases):
        #scipy.signal.convolve2d(matX, matY, mode)
        pre_sampled += convolve2d(hidden_groups[group], self.weight_groups[group], mode='full')

    #shared visible bias
    pre_sampled += self.visible_bias

    #rng is numpy.random.RandomState.normal(loc, scale, size)
    #It samples from normal distribution
    #loc is the mean
    #scale is the std deviation
    sampled = self.rng.normal(loc=pre_sampled, scale=1, size=self.visible_layer_shape)

    #returning both pre-sampled and sampled values respectively for now
    return [pre_sampled, sampled]

Am I understanding this all correctly? Are my uses of the mean correct in both the binary and Gaussian cases?

Thanks a lot!


1 Answer 1


Yep, looks like to got them correct.

Your first equation gives the probability that a hidden unit should be turned on, i.e. a number between (0 , 1). You then use that probability to obtain a sample, and then as you correctly say, use that sample as the beginning point for Gibbs sampling. An easy way to obtain a sample using the binary probabilities:

bin_sample = probs > np.random.uniform(low=0, high=1, size=[shape of probs])

The method you mention using a Bernoulli distribution is equivalent.

Your second equation specifies the probability distribution that a real valued Gaussian Unit should take values from. To obtain a sample here, use the calculated probability for each hidden unit as the mean along with a standard deviation of 1, and draw from a normal distribution.

gaus_sample = np.random.normal(loc=probs, scale=1, size=[shape of probs])

Look at 'END OF POSITIVE PHASE' here for an example from Geoffrey Hinton.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.