2
$\begingroup$

On page 12 of this tutorial, it shows that the learning rule is $\Delta w_{ji} = \epsilon s_j(s_i-p_i)$. Can someone show me how this is derived? I got $(1-p_i)s_j$ instead after computing $\frac{d\log p_i}{dw_{ji}}$, where $p_i=\frac{1}{1+\exp^{-\sum_j s_j w_{ji}}}$.

I noticed that this learning rule is different from the one in Neal's paper. Why is that?

Last question, is the difficulty of learning sigmoid belief nets due to the posterior $p_i$ in the learning rule?

$\endgroup$
2
$\begingroup$

I was looking around for some similar explanations and I believe I can help with this question.

What you think a mismatch is actually the same formula with a more general scope. Let us say our target unit is s_i and using the sigmoid function we define the probability of this target unit given the status of the hidden units. Thus, the evaluation of sigmoid() corresponds to the probability of the target unit being on. Now if you were asked to calculate the derivative of the probability of the unit activating wrt the weight of a specific connection you would calculate it as:

And as you explain the result would be

However if you were asked to compute the derivative of the probability that the target unit is switched off, you would use 1-sigmoid() and then when you apply the chain rule for derivation in the following expression

Now the result is . Now these two different derivaties correspond to the target unit taking the values 1 and 0 respectively so both expressions can be summarised as:

Now to generalise here you can find some more theoretical details:

SBN simple example

Gradient derivation rule

Regarding the paper from Neal's the formulation is the same, the only difference is that here we have made use of the following calculation for the sigmoid function derivative:

While he applies:

because

And finally to the last question I believe the difficulty lies in the fact that it is typically hard to compute the posterior distribution over all possible configurations of hidden causes. It is also hard to compute the probability of an observed vector and hence the definition of methods to train the network that rely on an estimation of the distribution combined with either minimising the energy of observed states on the generative model or approximations to MAP learning.

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.