I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=5eAXoPSBgnE)

The softmax activation function is $y_i = \frac{e^{x_i}}{\sigma_k e^{x_k}}$ and I have already computed the Jacobian as $J_{ij} = \frac{\partial y_i}{\partial x_j} = y_i ( \delta_{ij} - y_j)$

Now we are asked to show that $\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j s_j$ where $s_j = \frac{\partial L}{\partial y_j} y_j$

. .

Firstly, I do not like the fact that the first term on the RHS is vectorised whilst the second is not but I assume this is meant to be a constant that is subtracted from every element.

Anyway, here is my attempt so far:

$\frac{\partial L}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} \frac{\partial y_j}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} y_j \delta_{ij} - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$

where I've used the result I gave above for the Jacobian.

Now, if I compute the sum in the first term, the contraction over the Kronecker delta gives

$ = \frac{\partial L}{\partial y_i} y_i - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$

and if I revert to vector notation, I get

$\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j \frac{\partial L}{\partial y_j} y_j \vec{y} = \vec{s} - \sum_j s_j \vec{y}$

which seems almost correct except for that pesky $\vec{y}$ that is present in the final term. Any suggestions?


2 Answers 2


Define the variables $$\eqalign{ Y &= {\rm Diag}(y), \quad p = \frac{\partial L}{\partial y}, \quad s = Yp\cr }$$

Then $$\eqalign{ \frac{\partial y}{\partial x} &= \big(Y - yy^T\big) = \big(I-y1^T\big)Y \cr }$$ and $$\eqalign{ \frac{\partial L}{\partial x} &= \frac{\partial y}{\partial x}\,\,\frac{\partial L}{\partial y}\cr &= \big(I-y1^T\big)Yp \cr &= \big(I-y1^T\big)s \cr &= s - \big(1^Ts\big)y \cr }$$ which matches your result.

  • $\begingroup$ Thanks but again I am still confused by the y on the final term. As pointed out in the above comment, this is a slight notation issue. However the notation on the slide seems to suggest we should be subtracting this scalar term from every component of the vector term. $\endgroup$
    – user11128
    Jan 28, 2019 at 22:50
  • $\begingroup$ (cont) but the components of y will be decimal numbers so we may end up subtracting e.g. 0.5* scalar from first component and 2.638*scalar from second component etc. Do you see my point? $\endgroup$
    – user11128
    Jan 28, 2019 at 22:52
  • $\begingroup$ @user11128 No, I don't see your point. Since $(1^Ts)$ is a scalar, let's call that $\alpha$, and $\{s,y\}$ are vectors, all I see is the subtraction of two vectors, i.e. $(s-\alpha y)$. $\endgroup$
    – greg
    Jan 29, 2019 at 2:37
  • $\begingroup$ Me too. But there is no vector y in the second term in the original expression? $\endgroup$
    – user11128
    Jan 29, 2019 at 11:07
  • $\begingroup$ To recover the original equation set $y$ to a vector of all ones. $\endgroup$
    – greg
    Jan 29, 2019 at 21:49

The expression for the derivative $\frac{\partial L}{\partial \textbf{x}}=\mathbf{s}-\alpha$ is not actually defined in usual linear algebra. It's subtracting a scalar from a vector. They should be of same dimensions. However, if you code this in Python, R, Matlab, or C++ (via operator overloading etc.), it most probably won't give you any errors. So, programmatically, this expression is correct, but mathematically it is wrong, where it actually should have been. I think, the presenter wasn't saying "Check my math" in vain. Your derivations seems correct.

  • $\begingroup$ yes I get the scalar/vector issue. My concern is that my scalar is multiplying the vector $\vec{y}$ rather than a vector of ones? $\endgroup$
    – user11128
    Jan 28, 2019 at 9:54
  • $\begingroup$ @user11128 other than the scalar issue, as I said, your derivations seem correct. If you’re still unsure, best way is to check the numerical gradients. $\endgroup$
    – gunes
    Jan 28, 2019 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.