I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=5eAXoPSBgnE)

The softmax activation function is $y_i = \frac{e^{x_i}}{\sigma_k e^{x_k}}$ and I have already computed the Jacobian as $J_{ij} = \frac{\partial y_i}{\partial x_j} = y_i ( \delta_{ij} - y_j)$

Now we are asked to show that $\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j s_j$ where $s_j = \frac{\partial L}{\partial y_j} y_j$

. .

Firstly, I do not like the fact that the first term on the RHS is vectorised whilst the second is not but I assume this is meant to be a constant that is subtracted from every element.

Anyway, here is my attempt so far:

$\frac{\partial L}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} \frac{\partial y_j}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} y_j \delta_{ij} - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$

where I've used the result I gave above for the Jacobian.

Now, if I compute the sum in the first term, the contraction over the Kronecker delta gives

$ = \frac{\partial L}{\partial y_i} y_i - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$

and if I revert to vector notation, I get

$\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j \frac{\partial L}{\partial y_j} y_j \vec{y} = \vec{s} - \sum_j s_j \vec{y}$

which seems almost correct except for that pesky $\vec{y}$ that is present in the final term. Any suggestions?


Define the variables $$\eqalign{ Y &= {\rm Diag}(y), \quad p = \frac{\partial L}{\partial y}, \quad s = Yp\cr }$$

Then $$\eqalign{ \frac{\partial y}{\partial x} &= \big(Y - yy^T\big) = \big(I-y1^T\big)Y \cr }$$ and $$\eqalign{ \frac{\partial L}{\partial x} &= \frac{\partial y}{\partial x}\,\,\frac{\partial L}{\partial y}\cr &= \big(I-y1^T\big)Yp \cr &= \big(I-y1^T\big)s \cr &= s - \big(1^Ts\big)y \cr }$$ which matches your result.

| cite | improve this answer | |
  • $\begingroup$ Thanks but again I am still confused by the y on the final term. As pointed out in the above comment, this is a slight notation issue. However the notation on the slide seems to suggest we should be subtracting this scalar term from every component of the vector term. $\endgroup$ – user11128 Jan 28 '19 at 22:50
  • $\begingroup$ (cont) but the components of y will be decimal numbers so we may end up subtracting e.g. 0.5* scalar from first component and 2.638*scalar from second component etc. Do you see my point? $\endgroup$ – user11128 Jan 28 '19 at 22:52
  • $\begingroup$ @user11128 No, I don't see your point. Since $(1^Ts)$ is a scalar, let's call that $\alpha$, and $\{s,y\}$ are vectors, all I see is the subtraction of two vectors, i.e. $(s-\alpha y)$. $\endgroup$ – greg Jan 29 '19 at 2:37
  • $\begingroup$ Me too. But there is no vector y in the second term in the original expression? $\endgroup$ – user11128 Jan 29 '19 at 11:07
  • $\begingroup$ To recover the original equation set $y$ to a vector of all ones. $\endgroup$ – greg Jan 29 '19 at 21:49

The expression for the derivative $\frac{\partial L}{\partial \textbf{x}}=\mathbf{s}-\alpha$ is not actually defined in usual linear algebra. It's subtracting a scalar from a vector. They should be of same dimensions. However, if you code this in Python, R, Matlab, or C++ (via operator overloading etc.), it most probably won't give you any errors. So, programmatically, this expression is correct, but mathematically it is wrong, where it actually should have been. I think, the presenter wasn't saying "Check my math" in vain. Your derivations seems correct.

| cite | improve this answer | |
  • $\begingroup$ yes I get the scalar/vector issue. My concern is that my scalar is multiplying the vector $\vec{y}$ rather than a vector of ones? $\endgroup$ – user11128 Jan 28 '19 at 9:54
  • $\begingroup$ @user11128 other than the scalar issue, as I said, your derivations seem correct. If you’re still unsure, best way is to check the numerical gradients. $\endgroup$ – gunes Jan 28 '19 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.