2
$\begingroup$

When generating samples from a DBN, how do you handle the biases that have been learned for the layers below?

I know that you normally perform a number of Block Gibbs sampling steps in the undirected top 2 layers of the DBN, then propagate the sampled values of the penultimate layer down to the visible layer.

So when propagating downwards, do you also include the visible biases learned during greedy layer-wise training, when sampling a layer below given the states of the layer above?

Hope my question makes sense...

$\endgroup$
2
$\begingroup$

Yes, you also have to include the bias terms.

In general, when training a deep belief network, you will learn a distribution $p_1(\mathbf{v}, \mathbf{h}_1)$ when training the first layer, a distribution $p_2(\mathbf{h}_1, \mathbf{h}_2)$ when training the second layer, and so on. After training $N$ layers, you "stack" these distributions to yield the distribution of the deep belief network, $$p_1(\mathbf{v} \mid \mathbf{h}_1) \cdot \prod_{n = 2}^{N - 1} p_n(\mathbf{h}_{n - 1} \mid \mathbf{h}_n) \cdot p_N(\mathbf{h}_{N - 1}, \mathbf{h}_N).$$

The most straight-forward way to generate a sample from this distribution is to sample $p_N(\mathbf{h}_{N - 1}, \mathbf{h}_N)$, then $p_{N - 1}(\mathbf{h}_{N - 2} \mid \mathbf{h}_{N - 1})$, and lastly $p_1(\mathbf{v} \mid \mathbf{h}_1)$. In case of RBMs, the conditional distribution is given by (I use bold letters for vectors and non-bold letters for one-dimensional variables)

\begin{align} p_1(\mathbf{v} \mid \mathbf{h}_1) = \prod_i p_1(v_i \mid \mathbf{h}_1), \\ p_1(v_i = 1 \mid \mathbf{h}_1) = \sigma(\mathbf{w}_i^\top \mathbf{h}_1 + b_i). \end{align}

So, yes, the bias term needs to be included.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.