1
$\begingroup$

I am new to Neural Networks and trying to implement RBM. I am stuck on initializing the visible layer's bias value. Is it supposed to initialize to some random number or there is some probabilistic distribution by which it should be initialized. In python I am seeing:

> "vbias = theano.shared(value =numpy.zeros(n_visible,
>                               dtype = theano.config.floatX), name='vbias') "

but don't know what is happening here in this line.

$\endgroup$
1
$\begingroup$

That line of code is initializing the visible biases to zero. In Hinton's RBM guide (available here) he recommends initializing the bias of the $i^{th}$ visible to $$ b_i = \log\left(\frac{p_i}{1 - p_i}\right) $$ where $p_i$ the proportion of training instances where the $i^{th}$ binary feature is 1. The reasoning behind this is if you were to then sample from the visible units using just the biases as input you would have that the probability that the $i^{th}$ unit is on is given by $$ \begin{align*} P(v_i = 1) &= \frac{1}{1 + e^{-b_i}}\\ &= \frac{1}{1 + e^{-\log(p_i / (1 - p_i))}}\\ &= \frac{1}{1 + \frac{1 - p_i}{p_i}}\\ &= p_i. \end{align*} $$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ So the python code the OP posted is incorrect? Or at least: sub-optimal? Thanks for the link! I will most probably base my handling of RBMs on that guide. $\endgroup$ – Angelorf Mar 12 '14 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.