# What is the update rule for hidden layer if softmax activation function is used?

I am trying to understand how backpropagation works. I understood the basic concepts and became familiar with derivation of equations for sigmoid activation function. Specifically for hidden layers, for w1 (from picture below), the gradient looks like-

$$E_{total}=\frac{1}{2}(target_{o1}-out_{o1})^2+\frac{1}{2}(target_{o2}-out_{o2})^2$$

$$\frac{\partial{E_{total}}}{\partial{w_{1}}}=\sum_{i}{\frac{\partial{E_{total}}}{\partial{net_{hi}}}}*\frac{\partial{net_{hi}}}{\partial{w_{1}}}={\frac{\partial{E_{total}}}{\partial{net_{h1}}}}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}[if i\neq{1}, \frac{\partial{net_{hi}}}{\partial{w_{1}}}=0]$$ $$=\sum_{i}{(\frac{\partial{E_{total}}}{\partial{out_{hi}}}*\frac{\partial{out_{hi}}}{\partial{net_{h1}}})}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}$$ $$=\sum_{i}{(\sum_{j}(\frac{\partial{E_{total}}}{\partial{net_{oj}}}*\frac{\partial{net_{oj}}}{\partial{out_{hi}}})*\frac{\partial{out_{hi}}}{\partial{net_{h1}}})}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}$$ Now for sigmoid activation function if $i\neq{1} \frac{\partial{out_{hi}}}{\partial{net_{h1}}}=0$ so we get $$=\sum_{j}(\frac{\partial{E_{total}}}{\partial{net_{oj}}}*\frac{\partial{net_{oj}}}{\partial{out_{h1}}})*\frac{\partial{out_{h1}}}{\partial{net_{h1}}}*\frac{\partial{net_{h1}}}{\partial{w_{1}}}$$ $$=\sum_{j}(\delta_j*w_{oj})*(out_{h1})*(1-out_{h1})*i1$$ But for softmax activation function if $i\neq{1} \frac{\partial{out_{hi}}}{\partial{net_{h1}}}\neq{0}$ so we get $$=\sum_{i}\sum_{j}(\delta_j*w_{oj})*(out_{h1})*(1_{i=1}-out_{hi})*i1$$ for hidden layer. Is this derivation correct?

Also I am confused about the bias term. In sigmoid activation function it will not be a problem because $\sum_{i}out_{oi}$ can vary but for softmax $\sum_{i}out_{oi}=1$ (as fraction is being calculated). In that case how probabilistic interpretation for softmax layer holds true we include a unit whose value is 1 which is equal to $\sum_{i}out_{oi}$ (summation of all others units).

Also, for output layer I have shown my derivation in this question. Can someone verify? Thank you very much

• The seems like the same question you asked a few hours ago : stats.stackexchange.com/questions/243327/… Could you update your previous question instead of posting a new question? Oct 31, 2016 at 11:45
• This is not the same question @dimpol . As I had two questions, I divided it into two. In this question I am interested in derivation for hidden layers and how bias will be handled, in the previous question I was only concerned about output layer if summed squared error and softmax activation function are used.I also have given link to my other question. Oct 31, 2016 at 12:08
• You are right, as I started reading it, the question reminded me of your other question and I posted my comment without reading this question thoroughly. That was a mistake on my part, my apologies. Oct 31, 2016 at 12:13
• It's ok @dimpol , btw can you please verify if my derivation is correct. It would be a great help! Oct 31, 2016 at 12:18