1
$\begingroup$

I'm reading this tutorial (presented below) on computing derivative of crossentropy. The author used the loss function of logistic regression I think. https://www.dropbox.com/s/rxrtz3auu845fuy/Softmax.pdf?dl=0

Most of the equations make sense to me except one thing. In the second page, there is: $$\frac{\partial E_x}{\partial o^x_j}=\frac{t_j^x}{o_j^x}+\frac{1-t_j^x}{1-o^x_j}$$ However in the third page, the "Crossentropy derivative" becomes

$$\frac{\partial E_x}{\partial o^x_j}=-\frac{t_j^x}{o_j^x}+\frac{1-t_j^x}{1-o^x_j}$$

There is a minus sign in $E_x$. Then the derivative should be $\frac{\partial E_x}{\partial o^x_j}=-\frac{t_j^x}{o_j^x}-\frac{1-t_j^x}{1-o^x_j}$. But it is not. What have I missed?


The tutorial:

enter image description here enter image description here

$\endgroup$
2
$\begingroup$

There is indeed a mistake:\begin{align} \frac{\partial E_x}{\partial o_j^x} &=\frac{\partial }{\partial o_j^x} \left( - \sum_{k}[t_k^x \log(o_k^x)] + (1-t_k^x) \log(1-o_k^x)]\right) \\ &=-\frac{\partial }{\partial o_j^x} \left( \sum_{k}[t_k^x \log(o_k^x)] + (1-t_k^x) \log(1-o_k^x)]\right) \\ &=-\frac{\partial }{\partial o_j^x} \left( [t_j^x \log(o_j^x)] + (1-t_j^x) \log(1-o_j^x)]\right) \\ &=- \left( \frac{t_j^x}{o_j^x} - \frac{1-t_j^x}{1-o_j^x}\right), \text{Chain rule} \\ &=- \frac{t_j^x}{o_j^x} + \frac{1-t_j^x}{1-o_j^x} \\ \end{align}

$\endgroup$
0
$\begingroup$

An easy way to remember this is to internalize the gradient of the cross-entropy with respect to network parameters, which is famously $t_i - o_i$.

The last slide does this correctly. So, it looks like the second slide has a mistake. If you follow the derivations you'll notice the mistake where for no reason a minus sign appears in the middle of the right hand side (before the last equation).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.