# Compute Gradient of Cross Entropy Loss with respect to its logits

I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 section(7)

$$\hat{y} = softmax(\theta)$$

$$L = CrossEntropy(y, \hat{y})$$

Prove that: The gradient is $$∂L/∂θ = \hat{y} − y$$

My approach so far:

What we know is that softmax function is given as: $$\hat{y_i} = \frac{\exp^{x{_i}}}{\sum_{k}\exp^{x_{i}}}$$ eq.(1)

,where $$x$$=input value (in case of 1 layer (input, output).

Also $$L=-\sum y_i*\ln{\hat{y_i}}$$ eq.(2)

1. Derivative of softmax when $$i=j$$, (after applying the quotient rule of derivatives)

$$\frac{\theta{\hat{y_i}}}{\theta{x_i}} = \frac{ \sum{_k}\exp^{x_i}*\exp^{x_i} - \exp^{x_i}*\exp^{x_i} }{ (\sum_k{\exp^{x_i})^2} }$$ eq.(3)

Simplifying equation (3) using equation (1), we have

$$\frac{\theta{\hat{y_i}}}{\theta{x_i}}=\hat{y_i}(\frac{\sum{_k}\exp^{x_i}}{\sum{_k}\exp^{x_i}}-\hat{y_i}) = \hat{y_i}(1-\hat{y_i})$$ eq.(4)

1. Derivative of softmax when $$i\neq j$$,

$$\frac{\theta{\hat{y_i}}}{\theta{x_j}} = \frac{ 0 - \exp^{x_i}*\exp^{x_i} }{ (\sum_k{\exp^{x_i})^2} }$$ eq.(5)

Again equation (5) can be simplified using eq. (1)

$$\frac{\theta{\hat{y_i}}}{\theta{x_j}} = - \hat{y_i}*\hat{y_j}$$ eq.(6)

1. Derivative of loss function (cross-entropy) wrt to a chosen output

$$\frac{\theta L }{\theta \hat{y_i}} = - \sum{y_i*\frac{1}{\hat{y_i}}}$$ eq. (7)

1. Derivative of loss function (cross-entropy) wrt to a chosen input - Backpropagation

(multiplication rule - replacing eq. 7) $$\frac{\theta L }{\theta {x_j}} = \frac{\theta L }{\theta \hat{y_i}} * \frac{\theta \hat{y_i}}{\theta x_j} = - \sum_{i\neq{j}} y_i\frac{1}{\hat{y_i}}*\frac{\theta \hat{y_i}}{\theta x_j} - \sum_{i=j} y_i\frac{1}{\hat{y_i}}*\frac{\theta \hat{y_i}}{\theta x_i}$$

Replacing eq. 6, 4 in the equation above we get

$$\frac{\theta L }{\theta {x_j}} = - \sum_{i\neq{j}} y_i\frac{1}{\hat{y_i}} (- \hat{y_i}*\hat{y_j}) - \sum_{i=j}y_i\frac{1}{\hat{y_i}}\hat{y_i}(1-\hat{y_i}) = - \sum_{i\neq{j}} y_i\hat{y_j} + \sum_{i=j} y_i\hat{y_i} - y_j$$ eq. (8)

Now since we use softmax activation function $$\sum{y_i} = 1$$, so eq .8 is:

$$\frac{\theta L }{\theta {x_j}} = 1(\hat{y_j} - y_j)$$

Based on that approach I am not sure if I take into consideration the logits aka the scores of the model (vector of scores before applying the softmax). Could someone throw some light on this?

• What have you tried? Where are you stuck? Note that you can use math typesetting via mathjax. More information: math.meta.stackexchange.com/questions/5020/…
– Sycorax
Jan 13 at 19:56
• @Sycorax I have added my approach. I am new to the website. Much appreciated your notes. Jan 13 at 22:49
• yaroslavvb.medium.com/… Jan 14 at 16:48
• @YaroslavBulatov I would appreciate it if you explain why the posted article is related to my question and how could be the solution I am looking for. An explanatory answer would be very helpful. Thanks Jan 15 at 10:24
• It's an alternative way of deriving the gradient,.which may come useful after you learn the algebra method Jan 15 at 13:50