I am new to Neural Networks and trying to implement RBM. I am stuck on initializing the visible layer's bias value. Is it supposed to initialize to some random number or there is some probabilistic distribution by which it should be initialized. In python I am seeing:
> "vbias = theano.shared(value =numpy.zeros(n_visible,
> dtype = theano.config.floatX), name='vbias') "
but don't know what is happening here in this line.
That line of code is initializing the visible biases to zero. In Hinton's RBM guide (available here) he recommends initializing the bias of the $i^{th}$ visible to $$ b_i = \log\left(\frac{p_i}{1 - p_i}\right) $$ where $p_i$ the proportion of training instances where the $i^{th}$ binary feature is 1. The reasoning behind this is if you were to then sample from the visible units using just the biases as input you would have that the probability that the $i^{th}$ unit is on is given by $$ \begin{align*} P(v_i = 1) &= \frac{1}{1 + e^{-b_i}}\\ &= \frac{1}{1 + e^{-\log(p_i / (1 - p_i))}}\\ &= \frac{1}{1 + \frac{1 - p_i}{p_i}}\\ &= p_i. \end{align*} $$
