I am trying to understand how backpropagation works. I understood the basic concepts and became familiar with derivation of equations for sigmoid activation function. Specifically for hidden layers, for w1 (from picture below), the gradient looks like- enter image description here


$$\frac{\partial{E_{total}}}{\partial{w_{1}}}=\sum_{i}{\frac{\partial{E_{total}}}{\partial{net_{hi}}}}*\frac{\partial{net_{hi}}}{\partial{w_{1}}}={\frac{\partial{E_{total}}}{\partial{net_{h1}}}}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}[if i\neq{1}, \frac{\partial{net_{hi}}}{\partial{w_{1}}}=0]$$ $$=\sum_{i}{(\frac{\partial{E_{total}}}{\partial{out_{hi}}}*\frac{\partial{out_{hi}}}{\partial{net_{h1}}})}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}$$ $$=\sum_{i}{(\sum_{j}(\frac{\partial{E_{total}}}{\partial{net_{oj}}}*\frac{\partial{net_{oj}}}{\partial{out_{hi}}})*\frac{\partial{out_{hi}}}{\partial{net_{h1}}})}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}$$ Now for sigmoid activation function if $i\neq{1} \frac{\partial{out_{hi}}}{\partial{net_{h1}}}=0$ so we get $$=\sum_{j}(\frac{\partial{E_{total}}}{\partial{net_{oj}}}*\frac{\partial{net_{oj}}}{\partial{out_{h1}}})*\frac{\partial{out_{h1}}}{\partial{net_{h1}}}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}$$ $$=\sum_{j}(\delta_j*w_{oj})*(out_{h1})*(1-out_{h1})*i1$$ But for softmax activation function if $i\neq{1} \frac{\partial{out_{hi}}}{\partial{net_{h1}}}\neq{0}$ so we get $$=\sum_{i}\sum_{j}(\delta_j*w_{oj})*(out_{h1})*(1_{i=1}-out_{hi})*i1$$ for hidden layer. Is this derivation correct?

Also I am confused about the bias term. In sigmoid activation function it will not be a problem because $\sum_{i}out_{oi}$ can vary but for softmax $\sum_{i}out_{oi}=1$ (as fraction is being calculated). In that case how probabilistic interpretation for softmax layer holds true we include a unit whose value is 1 which is equal to $\sum_{i}out_{oi}$ (summation of all others units).

Also, for output layer I have shown my derivation in this question. Can someone verify? Thank you very much

  • $\begingroup$ The seems like the same question you asked a few hours ago : stats.stackexchange.com/questions/243327/… Could you update your previous question instead of posting a new question? $\endgroup$
    – dimpol
    Oct 31 '16 at 11:45
  • $\begingroup$ This is not the same question @dimpol . As I had two questions, I divided it into two. In this question I am interested in derivation for hidden layers and how bias will be handled, in the previous question I was only concerned about output layer if summed squared error and softmax activation function are used.I also have given link to my other question. $\endgroup$ Oct 31 '16 at 12:08
  • $\begingroup$ You are right, as I started reading it, the question reminded me of your other question and I posted my comment without reading this question thoroughly. That was a mistake on my part, my apologies. $\endgroup$
    – dimpol
    Oct 31 '16 at 12:13
  • $\begingroup$ It's ok @dimpol , btw can you please verify if my derivation is correct. It would be a great help! $\endgroup$ Oct 31 '16 at 12:18

It seems that your derivation is correct.

The bias part can be also treated as a variable whose input is always 1. The bias also has a parameter, so seldom can the production be 1.

I thought the Maximum likelihood(also is the cross entropy here) can be used as the objective function to improve the performance of the backpropagation. Please refer to this explanation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.