# Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=5eAXoPSBgnE)

The softmax activation function is $$y_i = \frac{e^{x_i}}{\sigma_k e^{x_k}}$$ and I have already computed the Jacobian as $$J_{ij} = \frac{\partial y_i}{\partial x_j} = y_i ( \delta_{ij} - y_j)$$

Now we are asked to show that $$\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j s_j$$ where $$s_j = \frac{\partial L}{\partial y_j} y_j$$

. .

Firstly, I do not like the fact that the first term on the RHS is vectorised whilst the second is not but I assume this is meant to be a constant that is subtracted from every element.

Anyway, here is my attempt so far:

$$\frac{\partial L}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} \frac{\partial y_j}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} y_j \delta_{ij} - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$$

where I've used the result I gave above for the Jacobian.

Now, if I compute the sum in the first term, the contraction over the Kronecker delta gives

$$= \frac{\partial L}{\partial y_i} y_i - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$$

and if I revert to vector notation, I get

$$\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j \frac{\partial L}{\partial y_j} y_j \vec{y} = \vec{s} - \sum_j s_j \vec{y}$$

which seems almost correct except for that pesky $$\vec{y}$$ that is present in the final term. Any suggestions?

Define the variables \eqalign{ Y &= {\rm Diag}(y), \quad p = \frac{\partial L}{\partial y}, \quad s = Yp\cr }

Then \eqalign{ \frac{\partial y}{\partial x} &= \big(Y - yy^T\big) = \big(I-y1^T\big)Y \cr } and \eqalign{ \frac{\partial L}{\partial x} &= \frac{\partial y}{\partial x}\,\,\frac{\partial L}{\partial y}\cr &= \big(I-y1^T\big)Yp \cr &= \big(I-y1^T\big)s \cr &= s - \big(1^Ts\big)y \cr } which matches your result.

• Thanks but again I am still confused by the y on the final term. As pointed out in the above comment, this is a slight notation issue. However the notation on the slide seems to suggest we should be subtracting this scalar term from every component of the vector term. – user11128 Jan 28 '19 at 22:50
• (cont) but the components of y will be decimal numbers so we may end up subtracting e.g. 0.5* scalar from first component and 2.638*scalar from second component etc. Do you see my point? – user11128 Jan 28 '19 at 22:52
• @user11128 No, I don't see your point. Since $(1^Ts)$ is a scalar, let's call that $\alpha$, and $\{s,y\}$ are vectors, all I see is the subtraction of two vectors, i.e. $(s-\alpha y)$. – greg Jan 29 '19 at 2:37
• Me too. But there is no vector y in the second term in the original expression? – user11128 Jan 29 '19 at 11:07
• To recover the original equation set $y$ to a vector of all ones. – greg Jan 29 '19 at 21:49

The expression for the derivative $$\frac{\partial L}{\partial \textbf{x}}=\mathbf{s}-\alpha$$ is not actually defined in usual linear algebra. It's subtracting a scalar from a vector. They should be of same dimensions. However, if you code this in Python, R, Matlab, or C++ (via operator overloading etc.), it most probably won't give you any errors. So, programmatically, this expression is correct, but mathematically it is wrong, where it actually should have been. I think, the presenter wasn't saying "Check my math" in vain. Your derivations seems correct.

• yes I get the scalar/vector issue. My concern is that my scalar is multiplying the vector $\vec{y}$ rather than a vector of ones? – user11128 Jan 28 '19 at 9:54
• @user11128 other than the scalar issue, as I said, your derivations seem correct. If you’re still unsure, best way is to check the numerical gradients. – gunes Jan 28 '19 at 10:00