Understanding the distribution of the Spike & Slab Restricted Boltzmann Machine (ssRBM)

Question

The ssRBM is described as a way to model mean and covariance using Restricted Boltzmann Machines.

I'm reading the paper that introduced the spike and slab restricted boltzmann machine. I have yet do more than skim the follow up paper that fine-tuned the model.

The variables involved are:

• $v$ = real valued continuous "data variable"

• $h \in \lbrace{ 0,1 \rbrace}^K$= binary hidden variable (spike)

• $s \in \mathbb{R}^K$ = continuous hidden variable (slab)

And the idea is that rather than just using the hidden variable $h$ as is done in the traditional RBM, $h$ is element-wise multiplied by $s$, so that one only has a subset of continuous random latent variables, $s \odot h$, likely active for a given input $v$.

The distribution function in the first of these papers (top of page 3 or "235") is written.

$$p(v,h,s) = \frac{1}{Z} e^{-\frac{1}{2}v^T\Lambda v \ +\ b^Th \ + \ \sum_i^N\big( v^TW_is_ih_i + \frac{1}{2}s_i^2\alpha_i\bigl)} \cdot \mathbb{U}(v;R)$$

I have two questions:

1. What is the purpose the uniform distribution over the "visible variables" or "data variables" $v$, $\mathbb{R}(v;R)$?

   The author(s) say

   $\mathbb{U}(v;R)$ represents a distribution that is uniform over a ball radius R, centered at the origin, that contains all the training data, i.e., $R > \max_t ||v_t||_2$ ($t$ indexes over training examples). The region of the visible layer space outside the ball has zero probability under the model. This restriction to a finite domain guarantees that the partition function $Z$ remains finite. We can think of the distribution presented in equations 2 and 1, as being associated with the bipartite graph structure of the RBM with the distinction that the hidden layer is composed of an element-wise product of the vectors $s$ and $h$.

   Why is this necessary? the exponential term alone of the distribution is Gaussian in $v$ so it is still well defined. After this paragraph, I don't see any other significant explanation or motivation. From my perspective this is an unnecessary complication to an already complete model.

2. It appears to me that the terms involving $W_i$ should be negative, otherwise the exponential term won't converge when integrated in $s$ as the distribution should be Gaussian in $s$; Is this a typo?

   The authors go on to derive conditional distributions with $s$ as Gaussian with mean and covariance parameters dependent on the other variables.

Answer 1

The partition function is a sum over all the possible combinations. As it says in the explanation "This restriction to a finite domain guarantees that the partition function $Z$ remains finite". The ball of radius $R $ is greater than the biggest $v$, that's defined as the $L_2$ norm which is essentially the magnitude of your vector.

So we're happy now that the data is bounded by this uniform (zero probability outside the ball). How would you normalize your joint or conditional using the partition function if you're summing over $v \in R^D, h \in \{0,1\}^N$ as well as $s \in R^N$? In the conditional equation in the paper see how he has to solve the integral over the space of v in order to get the analytical form for the conditional. That wouldn't be possible if you wouldn't bound $v$. For a simple Gaussian RBM you can compute the conditional under unbounded $v$ space, but adding $s$ changes your problem.