Softmax layer derivative by hand

Question

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=5eAXoPSBgnE)

The softmax activation function is $y_i = \frac{e^{x_i}}{\sigma_k e^{x_k}}$ and I have already computed the Jacobian as $J_{ij} = \frac{\partial y_i}{\partial x_j} = y_i ( \delta_{ij} - y_j)$

Now we are asked to show that $\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j s_j$ where $s_j = \frac{\partial L}{\partial y_j} y_j$

. .

Firstly, I do not like the fact that the first term on the RHS is vectorised whilst the second is not but I assume this is meant to be a constant that is subtracted from every element.

Anyway, here is my attempt so far:

$\frac{\partial L}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} \frac{\partial y_j}{\partial x_i} = \sum_j \frac{\partial L}{\partial y_j} y_j \delta_{ij} - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$

where I've used the result I gave above for the Jacobian.

Now, if I compute the sum in the first term, the contraction over the Kronecker delta gives

$ = \frac{\partial L}{\partial y_i} y_i - \sum_j \frac{\partial L}{\partial y_j} y_j y_i$

and if I revert to vector notation, I get

$\frac{\partial L}{\partial \vec{x}} = \vec{s} - \sum_j \frac{\partial L}{\partial y_j} y_j \vec{y} = \vec{s} - \sum_j s_j \vec{y}$

which seems almost correct except for that pesky $\vec{y}$ that is present in the final term. Any suggestions?

Answer 1

Define the variables $$\eqalign{ Y &= {\rm Diag}(y), \quad p = \frac{\partial L}{\partial y}, \quad s = Yp\cr }$$

Then $$\eqalign{ \frac{\partial y}{\partial x} &= \big(Y - yy^T\big) = \big(I-y1^T\big)Y \cr }$$ and $$\eqalign{ \frac{\partial L}{\partial x} &= \frac{\partial y}{\partial x}\,\,\frac{\partial L}{\partial y}\cr &= \big(I-y1^T\big)Yp \cr &= \big(I-y1^T\big)s \cr &= s - \big(1^Ts\big)y \cr }$$ which matches your result.

Answer 2

The expression for the derivative $\frac{\partial L}{\partial \textbf{x}}=\mathbf{s}-\alpha$ is not actually defined in usual linear algebra. It's subtracting a scalar from a vector. They should be of same dimensions. However, if you code this in Python, R, Matlab, or C++ (via operator overloading etc.), it most probably won't give you any errors. So, programmatically, this expression is correct, but mathematically it is wrong, where it actually should have been. I think, the presenter wasn't saying "Check my math" in vain. Your derivations seems correct.