I am trying to confirm whether or not I am understanding the process described in the title.
I understand sampling at the hidden layer (Binary Units) to involve:
- For each group of hidden units, for each hidden unit find the probability of it being "on", given the input v:
where *v is the "valid" convolution - valid convlution of the transpose of group ks weight matrix with the input
- Now to get the final Sample, for each hidden unit, you sample once from a Binomial Distribution with std dev = 1 and mean = the hidden unit's probability of being "on" - if this sampled value < the probability of the unit being "on", set the value of the hidden unit = 1
This creates the "Sampled" value of the hidden layers - which will then be used as the starting point in the chain of Gibbs sampling in the Contrastive Divergence algorithm.
Now sampling at the visible layer (Gaussian Real units)
The papers listed define the probability of each visible unit being "on", given a group of Hidden Unit binary vectors:
where *f is the "full" convolution - the "full" convolution of group ks weight matrix with the kth group of hidden unit values
They do not elaborate on the function
Does this mean Sample the value from a Normal distribution with mean = x and std dev = y?
Does this also mean then that the probability of a visible unit being on IS the sample of that visible unit?
Ive added a Python function below to hopefully explain how I believe this sampling is defined to work in a less ambiguous way:
def sample_v_given_h(self, hidden_groups): pre_sampled = np.zeros(self.visible_layer_shape) for group in range(1, self.numBases): #scipy.signal.convolve2d(matX, matY, mode) pre_sampled += convolve2d(hidden_groups[group], self.weight_groups[group], mode='full') #shared visible bias pre_sampled += self.visible_bias #rng is numpy.random.RandomState.normal(loc, scale, size) #It samples from normal distribution #loc is the mean #scale is the std deviation sampled = self.rng.normal(loc=pre_sampled, scale=1, size=self.visible_layer_shape) #returning both pre-sampled and sampled values respectively for now return [pre_sampled, sampled]
Am I understanding this all correctly? are my uses of the mean correct in both the binary and gaussian cases?
Thanks a lot!