2
$\begingroup$

I am trying to confirm whether or not I am understanding the process described in the title.

I am implementing a CRMB (with Real Valued Gaussian Visible units and Binary hidden units) as outlined in Lee et al (2009b), explained deeper in Lee et al (2009a)

I understand sampling at the hidden layer (Binary Units) to involve:

  • For each group of hidden units, for each hidden unit find the probability of it being "on", given the input v:

enter image description here

where *v is the "valid" convolution - valid convlution of the transpose of group ks weight matrix with the input

  • Now to get the final Sample, for each hidden unit, you sample once from a Binomial Distribution with std dev = 1 and mean = the hidden unit's probability of being "on" - if this sampled value < the probability of the unit being "on", set the value of the hidden unit = 1

This creates the "Sampled" value of the hidden layers - which will then be used as the starting point in the chain of Gibbs sampling in the Contrastive Divergence algorithm.

Now sampling at the visible layer (Gaussian Real units)

The papers listed define the probability of each visible unit being "on", given a group of Hidden Unit binary vectors:

enter image description here

where *f is the "full" convolution - the "full" convolution of group ks weight matrix with the kth group of hidden unit values

They do not elaborate on the function

Normal(x,y)

Does this mean Sample the value from a Normal distribution with mean = x and std dev = y?

Does this also mean then that the probability of a visible unit being on IS the sample of that visible unit?

Ive added a Python function below to hopefully explain how I believe this sampling is defined to work in a less ambiguous way:

def sample_v_given_h(self, hidden_groups):
    pre_sampled = np.zeros(self.visible_layer_shape)
    for group in range(1, self.numBases):
        #scipy.signal.convolve2d(matX, matY, mode)
        pre_sampled += convolve2d(hidden_groups[group], self.weight_groups[group], mode='full')

    #shared visible bias
    pre_sampled += self.visible_bias

    #rng is numpy.random.RandomState.normal(loc, scale, size)
    #It samples from normal distribution
    #loc is the mean
    #scale is the std deviation
    sampled = self.rng.normal(loc=pre_sampled, scale=1, size=self.visible_layer_shape)

    #returning both pre-sampled and sampled values respectively for now
    return [pre_sampled, sampled]

Am I understanding this all correctly? are my uses of the mean correct in both the binary and gaussian cases?

Thanks a lot!

$\endgroup$
0
$\begingroup$

Yep, looks like to got them correct.

Your first equation gives the probability that a hidden unit should be turned on, i.e. a number between (0 , 1). You then use that probability to obtain a sample, and then as you correctly say, use that sample as the beginning point for Gibbs sampling. An easy way to obtain a sample using the binary probabilities:

bin_sample = probs > np.random.uniform(low=0, high=1, size=[shape of probs])

The method you mention using a Bernoulli distribution is equivalent.

Your second equation specifies the probability distribution that a real valued Gaussian Unit should take values from. To obtain a sample here, use the calculated probability for each hidden unit as the mean along with a standard deviation of 1, and draw from a normal distribution.

gaus_sample = np.random.normal(loc=probs, scale=1, size=[shape of probs])

Look at 'END OF POSITIVE PHASE' here for an example from Geoffrey Hinton.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.