# The learning rule for sigmoid belief nets

On page 12 of this tutorial, it shows that the learning rule is $\Delta w_{ji} = \epsilon s_j(s_i-p_i)$. Can someone show me how this is derived? I got $(1-p_i)s_j$ instead after computing $\frac{d\log p_i}{dw_{ji}}$, where $p_i=\frac{1}{1+\exp^{-\sum_j s_j w_{ji}}}$.

I noticed that this learning rule is different from the one in Neal's paper. Why is that?

Last question, is the difficulty of learning sigmoid belief nets due to the posterior $p_i$ in the learning rule?

I was looking around for some similar explanations and I believe I can help with this question.

What you think a mismatch is actually the same formula with a more general scope. Let us say our target unit is $s_i$ and using the sigmoid function we define the probability of this target unit given the status of the hidden units. Thus, the evaluation of sigmoid($\sum s_j_i\cdot w_j_i$) corresponds to the probability of the target unit being on. Now if you were asked to calculate the derivative of the probability of the unit activating wrt the weight of a specific connection you would calculate it as:

$\frac{\partial{log(\mathit{p(s_i=1)})} }{\partial w_j_i}$

And as you explain the result would be $(1-\mathit{p_i})\cdot s_j$

However if you were asked to compute the derivative of the probability that the target unit is switched off, you would use 1-sigmoid($\sum s_j_i\cdot w_j_i$) and then when you apply the chain rule for derivation in the following expression

$\frac{\partial{log(\mathit{p(s_i=0)})} }{\partial w_j_i}$

Now the result is $(-\mathit{p_i})\cdot s_j$. Now these two different derivaties correspond to the target unit $s_i$ taking the values 1 and 0 respectively so both expressions can be summarised as:

$\frac{\partial{log(\mathit{p(s_i=s_i{}')})} }{\partial w_j_i}=(s_i{}'-\mathit{p_i})\cdot s_j$

Now to generalise here you can find some more theoretical details:

Regarding the paper from Neal's the formulation is the same, the only difference is that here we have made use of the following calculation for the sigmoid function derivative:

$\sigma {}'(x)=\sigma (x)\cdot \(1- \sigma (x))$

While he applies:

$\sigma {}'(x)=\sigma (x)\cdot \sigma (-x)$

because

$\sigma (-x)=\(1- \sigma (x))$

And finally to the last question I believe the difficulty lies in the fact that it is typically hard to compute the posterior distribution over all possible configurations of hidden causes. It is also hard to compute the probability of an observed vector and hence the definition of methods to train the network that rely on an estimation of the distribution combined with either minimising the energy of observed states on the generative model or approximations to MAP learning.

Hope this helps!