Softmax layer derivative by hand I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=5eAXoPSBgnE)
The softmax activation function is $y_i = \frac{e^{x_i}}{\sigma_k e^{x_k}}$ and I have already computed the Jacobian as $J_{ij} = \frac{\partial y_i}{\partial x_j} = y_i ( \delta_{ij} - y_j)$
Now we are asked to show that $\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j s_j$ where $s_j = \frac{\partial L}{\partial y_j} y_j$
.
.
Firstly, I do not like the fact that the first term on the RHS is vectorised whilst the second is not but I assume this is meant to be a constant that is subtracted from every element.
Anyway, here is my attempt so far:
$\frac{\partial L}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} \frac{\partial y_j}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} y_j \delta_{ij} - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$
where I've used the result I gave above for the Jacobian. 
Now, if I compute the sum in the first term, the contraction over the Kronecker delta gives
$ = \frac{\partial L}{\partial y_i} y_i - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$
and if I revert to vector notation, I get
$\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j \frac{\partial L}{\partial y_j} y_j \vec{y} = \vec{s} - \sum_j s_j \vec{y}$
which seems almost correct except for that pesky $\vec{y}$ that is present in the final term. Any suggestions?
 A: Define the variables
$$\eqalign{
 Y &= {\rm Diag}(y), \quad
 p = \frac{\partial L}{\partial y}, \quad
 s = Yp\cr
}$$
Then
$$\eqalign{
\frac{\partial y}{\partial x} &= \big(Y - yy^T\big) = \big(I-y1^T\big)Y \cr
}$$
and
$$\eqalign{
\frac{\partial L}{\partial x}
 &= \frac{\partial y}{\partial x}\,\,\frac{\partial L}{\partial y}\cr
 &= \big(I-y1^T\big)Yp \cr
 &= \big(I-y1^T\big)s \cr
 &= s - \big(1^Ts\big)y \cr
}$$
which matches your result.
A: The expression for the derivative $\frac{\partial L}{\partial \textbf{x}}=\mathbf{s}-\alpha$ is not actually defined in usual linear algebra. It's subtracting a scalar from a vector. They should be of same dimensions. However, if you code this in Python, R, Matlab, or C++ (via operator overloading etc.), it most probably won't give you any errors. So, programmatically, this expression is correct, but mathematically it is wrong, where it actually should have been. I think, the presenter wasn't saying "Check my math" in vain. Your derivations seems correct.
