I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 section(7)

$\hat{y} = softmax(\theta)$

$L = CrossEntropy(y, \hat{y})$

Prove that: The gradient is $∂L/∂θ = \hat{y} − y$

My approach so far:

What we know is that softmax function is given as: $\hat{y_i} = \frac{\exp^{x{_i}}}{\sum_{k}\exp^{x_{i}}}$ eq.(1)

,where $x$=input value (in case of 1 layer (input, output).

Also $L=-\sum y_i*\ln{\hat{y_i}}$ eq.(2)

  1. Derivative of softmax when $i=j$, (after applying the quotient rule of derivatives)

$\frac{\theta{\hat{y_i}}}{\theta{x_i}} = \frac{ \sum{_k}\exp^{x_i}*\exp^{x_i} - \exp^{x_i}*\exp^{x_i} }{ (\sum_k{\exp^{x_i})^2} }$ eq.(3)

Simplifying equation (3) using equation (1), we have

$\frac{\theta{\hat{y_i}}}{\theta{x_i}}=\hat{y_i}(\frac{\sum{_k}\exp^{x_i}}{\sum{_k}\exp^{x_i}}-\hat{y_i}) = \hat{y_i}(1-\hat{y_i})$ eq.(4)

  1. Derivative of softmax when $i\neq j$,

$\frac{\theta{\hat{y_i}}}{\theta{x_j}} = \frac{ 0 - \exp^{x_i}*\exp^{x_i} }{ (\sum_k{\exp^{x_i})^2} }$ eq.(5)

Again equation (5) can be simplified using eq. (1)

$ \frac{\theta{\hat{y_i}}}{\theta{x_j}} = - \hat{y_i}*\hat{y_j} $ eq.(6)

  1. Derivative of loss function (cross-entropy) wrt to a chosen output

$ \frac{\theta L }{\theta \hat{y_i}} = - \sum{y_i*\frac{1}{\hat{y_i}}}$ eq. (7)

  1. Derivative of loss function (cross-entropy) wrt to a chosen input - Backpropagation

(multiplication rule - replacing eq. 7) $ \frac{\theta L }{\theta {x_j}} = \frac{\theta L }{\theta \hat{y_i}} * \frac{\theta \hat{y_i}}{\theta x_j} = - \sum_{i\neq{j}} y_i\frac{1}{\hat{y_i}}*\frac{\theta \hat{y_i}}{\theta x_j} - \sum_{i=j} y_i\frac{1}{\hat{y_i}}*\frac{\theta \hat{y_i}}{\theta x_i} $

Replacing eq. 6, 4 in the equation above we get

$ \frac{\theta L }{\theta {x_j}} = - \sum_{i\neq{j}} y_i\frac{1}{\hat{y_i}} (- \hat{y_i}*\hat{y_j}) - \sum_{i=j}y_i\frac{1}{\hat{y_i}}\hat{y_i}(1-\hat{y_i}) = - \sum_{i\neq{j}} y_i\hat{y_j} + \sum_{i=j} y_i\hat{y_i} - y_j $ eq. (8)

Now since we use softmax activation function $\sum{y_i} = 1$, so eq .8 is:

$ \frac{\theta L }{\theta {x_j}} = 1(\hat{y_j} - y_j) $

Based on that approach I am not sure if I take into consideration the logits aka the scores of the model (vector of scores before applying the softmax). Could someone throw some light on this?

  • $\begingroup$ What have you tried? Where are you stuck? Note that you can use math typesetting via mathjax. More information: math.meta.stackexchange.com/questions/5020/… $\endgroup$
    – Sycorax
    Jan 13 at 19:56
  • $\begingroup$ @Sycorax I have added my approach. I am new to the website. Much appreciated your notes. $\endgroup$
    – BDEngineer
    Jan 13 at 22:49
  • $\begingroup$ yaroslavvb.medium.com/… $\endgroup$ Jan 14 at 16:48
  • $\begingroup$ @YaroslavBulatov I would appreciate it if you explain why the posted article is related to my question and how could be the solution I am looking for. An explanatory answer would be very helpful. Thanks $\endgroup$
    – BDEngineer
    Jan 15 at 10:24
  • $\begingroup$ It's an alternative way of deriving the gradient,.which may come useful after you learn the algebra method $\endgroup$ Jan 15 at 13:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.