0
$\begingroup$

Consider the step by step backpropagation shown in this article. The neural network given there is:
enter image description here

The outer layer employs sigmoid. It also employs mean squared error loss function. During back-propagation, it calculates following gradient:

$$\frac{\partial E_{total}}{\partial w_5} = \color{red}{\frac{\partial E_{total}}{\partial out_{o1}}} \times \color{green}{\frac{\partial out_{o1}}{\partial in_{o1}}} \times \color{blue}{\frac{\partial in_{o1}}{\partial w_5}}$$ $$=\color{red}{-(y_{o1}-out_{o1})} \times \color{green}{out_{o1}(1-out_{01})} \times \color{blue}{out_{h1}}$$

Now, for softmax function $y$, $$\frac{\partial y_i}{\partial s_k} =\begin{cases} \color{green}{y_i(1-y_i)}, & \text{if $i=k$} \\ -y_iy_k, & \text{if $i\neq k$} \end{cases} $$

I am confused with what will happen when outer layer employes softmax. Will the partial derivative remain same as before:

$$\frac{\partial E_{total}}{\partial w_5} = \color{red}{\frac{\partial E_{total}}{\partial out_{o1}}} \times \color{green}{\frac{\partial out_{o1}}{\partial in_{o1}}} \times \color{blue}{\frac{\partial in_{o1}}{\partial w_5}}$$ $$=\color{red}{-(y_{o1}-out_{o1})} \times \color{green}{out_{o1}(1-out_{01})} \times \color{blue}{out_{h1}}$$

because, $i=k=o1$ in $\color{green}{\frac{\partial out_{o1}}{\partial in_{o1}}}$ or the term $-y_i y_k$ also appear somewhere? I am confused, because this video shows how $-y_iy_k$ should also appear, at the end, say at time 16:44:

enter image description here

PS: Sorry if cropped image at the end looks bad, I just wanted to point to what I was talking about in the video.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.