# Contrastive Divergence: making hidden states binary

When learning a Restricted Bolzmann Machine with Contrastive Divergence $\mathbf{CD}_n$ for $n>1$ should we make the hidden states binary in the updates that are in between?

In Hinton's - 'A Practical Guide to Training Restricted Bolzmann Machines' two contradictory reccomendations are mentioned:

3.1 "It is very important to make these hidden states binary, rather than using the probabilities themselves. [...] For the last update of the hidden units, [...] use the probability itself"

3.4 "When [the hidden units] are being driven by reconstructions, always use probabilities without sampling."

The first quote seems to say the updates in the middle should do sampling (making binary), while the second seems to say they shouldn't! Which one is correct?

## 2 Answers

I implemented it that way:

1. Sampling hidden from the real data, the data should be binary, therefore, it is not an issue, we take binary. It is the very first step of CD.
2. Sampling visible from hidden: Take the binary values (sampled values) of the hidden units.
3. Sampling hidden from visible: Take the activation probabilities of the visible units.

Step 2 and 3 are repeated if you use more than CD-1.

The quote "For the last update of the hidden units, [...] use the probability itself" means that for the very last update of the hidden units, you don't have to sample the values because it won't be used after, it is for efficiency.

• I implemented the other possibility. But which one is the right one? – Angelorf Jul 6 '14 at 9:29
• @Angelorf Have you good results for instance on MNIST reconstruction ? – Baptiste Wicht Jul 7 '14 at 6:30
• As I remembered CD-1 worked best, but for CD-1 there is no difference between the two possible CD algorithms. I would have to rerun my by now outdated code on MNIST. Maybe I will find time for this, but I am currently occupied with another feature extraction technique. – Angelorf Jul 8 '14 at 8:02

I have myself come to a very similar question, and I think i do now understand whats going on. I'll try to explain... :) I found  quite helpfull to come to my conclusions. I also want to stress, that I'm not entirely sure about the stuff that follows, so please correct me if I'm wrong.

Assume you do the followin sampling:

$$\mathbf{v^0} \rightarrow \mathbf{h^0} \rightarrow \mathbf{v^1}$$

where $\mathbf{v_0}$ is a visible binary data vector from your training set $\mathcal{T}$, $\mathbf{h^0}$ is the first sampled hidden binary vector and $\mathbf{v^1}$ is the first sampled visible vector (sampled from binary $\mathbf{h^0}$ and NOT its activations).

For a parameter update $\Delta w$ (taking CD1) of the weight matrix you usually write:

$$\Delta w \propto \left<\mathbf{h^0}\mathbf{v^0}\right>_\mathcal{T} - \left<\mathbf{h^0}\mathbf{v^1}\right>_\mathcal{T}$$

In  they rewrite the first bracket $\left<\mathbf{h^0}\mathbf{v^0}\right>_\mathcal{T}$ as $\left<\left<\mathbf{h^0}\mathbf{v^0}\right>_{p(\mathbf{h^0}|\mathbf{v^0},w)}\right>_\mathcal{T}$. The inner brackets average over the distribution of sampled hidden vectors for a specific data vector $p(\mathbf{h^0}|\mathbf{v^0},w)$ from the training set. This distribution is given by the activation probabilities for sampling $\mathbf{h_0}$ (lets call them $\mathbf{a_h^0}$) , hence you can directly perform the inner average by taking $\mathbf{a_h^0} \mathbf{v^0}$.

I think for the second bracket, one can write: $$\left<\mathbf{h^0}\mathbf{v^1}\right>_\mathcal{T} = \left<\left<\left<\mathbf{h^0}\mathbf{v^1}\right>_{p(\mathbf{v^1}|\mathbf{h^0},w)}\right>_{p(\mathbf{h^0}|\mathbf{v^0},w)}\right>_\mathcal{T}$$.

Here we can only perform the most inner average by taking the activations of the visible layer $\mathbf{a_v^1}$. One can easyly see this from the following (onedimensional) example.

Lets assume the probability to be active (h or v equals 1) is given by a sigmoid function:

$$P(h = 1 | v) = \mathcal{S}(v) \\ P(v = 1 | h) = \mathcal{S}(h)$$

the expectation values then read:

$$\left<h v\right>_{p(h|v)} = 1\times v\times \mathcal{S}(v) + 0 \times v \times (1-\mathcal{S}(v)) = v \times \mathcal{S}(v)$$

whereas for the second bracket one obtains:

$$\left<\left<h v\right>_{p(v|h)}\right>_{p(h|input)} = \left<h \times \mathcal{S}(h)\right>_{p(h|input)} = \mathcal{S}(1)\times \mathcal{S}(input) \neq \mathcal{S}(a_h)\times \mathcal{S}(input)$$.

The essence:

• For $CDn$ with $n>1$ we always have several nested expectation values.

• We can only replace the very last sampling of the Gibbs chain by their corresponding activations.

I think the above should explain

3.1 "It is very important to make these hidden states binary, rather than using the probabilities themselves. [...] For the last update of the hidden units, [...] use the probability itself"

When doing reconstructions, one usually does only one step of propagation ($v\rightarrow h$ or $h\rightarrow v$). Hence one can directly use the activations (i.e. probabilities) to get an expectation value of the sampling error.

As stated above, all of the above might be wrong. Therefore, I would be really glad if someone with more insight could comment on this explanation.

: Carreira-Perpinan, Miguel A., and Geoffrey E. Hinton. "On contrastive divergence learning." Proceedings of the tenth international workshop on artificial intelligence and statistics. NP: Society for Artificial Intelligence and Statistics, 2005.

• Oh my gosh, im not entirely sure anymore about: Here we can only perform the most inner average by taking the activations of the visible layer $\mathbf{a_v^1}$ because for binary units, the probability density collapses to delta peaks, and hence, one perfrom both average brackets... – Marti Nito Feb 26 '15 at 12:58
• Your answer only considers CD-1, while my question only arises with CD-n for $n>1$ – Angelorf Feb 27 '15 at 8:56
• @Angelorf I tried to be more specific wrt to CDn now, i hope you get the point of my argumentation now! I would really love to have another opinion to the whole issue. – Marti Nito Mar 2 '15 at 15:26
• Btw. i think my first comment was stupid, the delta function argument is stupid... – Marti Nito Mar 2 '15 at 15:27