I have been trying to create a program for training Neural Networks on my computer. For the Network in question, I have decided to use the Cross Entropy Error function:

$$E = -\sum_jt_j\ln o_j$$

Where $t_j$ is the target output for the Neuron $j$, and $o_j$ is the output of that neuron, attempting to predict $t_j$.

I want to know what $\frac{\delta E}{\delta o_j}$ is for some Neuron $j$. My intuition (plus my limited knowledge of calculus) lead me to believe that this value should be $-\frac{t_j}{o_j}$.

However, this does not seem to be correct. Cross Entropy is often used in tandem with the softmax function, such that $$o_j = \frac{e^{z_j}}{\sum_ke^{z_k}}$$ where z is the set of inputs to all neurons in the softmax layer (see here).

From this file, I gather that: $$\frac{\delta o_j}{\delta z_j} = o_j(1 - o_j)$$

According to this question: $$\frac{\delta E}{\delta z_j} = t_j - o_j$$ But this conflicts with my earlier guess of $\frac{\delta E}{\delta o_j}$. Why?

$$\frac{\delta E_j}{\delta z_j}=\frac{\delta E_j}{\delta o_j}\frac{\delta o_j}{\delta z_j}$$$$\Rightarrow\frac{\delta E_j}{\delta o_j}=\frac{\delta E_j}{\delta z_j}\div\frac{\delta o_j}{\delta z_j}$$ $$= \frac{t_j-o_j}{o_j(1-o_j)}$$ in direct contradiction to my earlier solution of $$-\frac{t_j}{o_j}$$ So which (if either) solution to $\frac{\delta E_j}{\delta o_j}$ is correct, and why?


1 Answer 1


Your $\frac{\partial E}{\partial o_j}$ is correct, but $\frac{\partial E}{\partial z_j}$ should be $$\frac{\partial E}{\partial z_j}=\sum_i\frac{\partial E}{\partial o_i}\frac{\partial o_i}{\partial z_j}$$ when $i=j$, using the results given in the post we have $$\frac{\partial E}{\partial o_j}\frac{\partial o_j}{\partial z_j}=-\frac{t_j}{o_j}o_j(1-o_j)=t_jo_j-t_j$$ when $i\neq j$ $$\frac{\partial o_i}{\partial z_j}=\frac{\partial \frac{e^{z_i}}{\sum_ke^{z_k}}}{\partial z_j}=-\frac{e^{z_i}}{(\sum_ke^{z_k})^2}e^{z_j}=-o_io_j$$ $$\frac{\partial E}{\partial o_i}\frac{\partial o_i}{\partial z_j}=-\frac{t_i}{o_i}(-o_io_j)=t_io_j$$ so the summation is $$\frac{\partial E}{\partial z_j}=\sum_i\frac{\partial E}{\partial o_i}\frac{\partial o_i}{\partial z_j}=\sum_it_io_j-t_j$$ since $t$ is a one-hot vector, $\sum_it_i=1$ therefore $$\frac{\partial E}{\partial z_j}=o_j-t_j$$ also see this question.

  • 1
    $\begingroup$ This answer helped me a lot. I made the same mistake, because in the case of regression the derivative: $$\frac{\partial E}{\partial z_j}=\sum_i\frac{\partial E}{\partial o_i}\frac{\partial o_i}{\partial z_j}$$ is reduced to: $$\frac{\partial E}{\partial z_j}=\frac{\partial E}{\partial o_j}\frac{\partial o_j}{\partial z_j}$$ But it does not apply in the case of classification. $\endgroup$
    – z_tjona
    Commented Aug 2, 2023 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.