How to find derivative of softmax function for the purpose of gradient descent?

Question

I'm trying to understand back propagation algorithm for multiclass classification using gradient descent. I'm using https://www.cs.toronto.edu/~graves/phd.pdf . The output layer is a softmax layer, in which each unit in that layer has activation function:

Here, a_k is the sum of inputs to unit 'k'.

Differentiating the above equation, the author has achieved this result.

I'm confused by the delta kk' and i have never seen anything like it.

Another question is do we consider the summation while taking the derivative, why or why not?

https://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function is a bit relevant, but the result of differentiation is different.

Answer 1

As whuber points out, $\delta_{ij}$ is the Kronecker delta https://en.wikipedia.org/wiki/Kronecker_delta :

$$ \begin{align} \delta_{ij} = \begin{cases} 0\: \text{when } i \ne j \\ 1 \: \text{when } i = j \end{cases} \end{align} $$

... and remember that a softmax has multiple inputs, a vector of inputs; and also gives a vector output, where the length of the input and output vectors are identical.

Each of the values in the output vector will change if any of the input vector values change. So the output vector values are each a function of all the input vector value:

$$ y_{k'} = f_{k'}(a_1, a_2, a_3,\dots, a_K) $$

where $k'$ is the index into the output vector, the vectors are of length $K$, and $f_{k'}$ is some function. So, the input vector is length $K$ and the output vector is length $K$, and both $k$ and $k'$ take values $\in \{1,2,3,...,K\}$.

When we differentiate $y_{k'}$, we differentiate partially with respect to each of the input vector values. So we will have:

• $\frac{\partial y_{k'}}{\partial a_1}$
• $\frac{\partial y_{k'}}{\partial a_2}$
• etc ...

Rather than calculating individually for each $a_1$, $a_2$ etc, we'll just use $k$ to represent the 1,2,3, etc, ie we will calculate:

$$ \frac{\partial y_{k'}}{\partial a_k} $$

...where:

• $k \in \{1,2,3,\dots,K\}$ and
• $k' \in \{1,2,3\dots K\}$

When we do this differentiation, eg see https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/ , the derivative will be:

$$ \frac{\partial y_{k'}}{\partial a_k} = \begin{cases} y_k(1 - y_{k'}) &\text{when }k = k'\\ - y_k y_{k'} &\text{when }k \ne k' \end{cases} $$

We can then write this using the Kronecker delta, which is simply for notational convenience, to avoid having to write out the 'cases' statement each time.

Answer 2

The author's formula violates the Einstein convention, since repeated indices imply summation.

A better way to write the result is $$ \frac{\partial y}{\partial a} = {\rm Diag}(y) - yy^T $$ where the Diag function creates a diagonal matrix by putting the vector $y$ along the main diagonal and zeros elsewhere.


If you wish to use the summation convention, you'll need to define a third-order tensor $T_{ijk}$ whose elements are equal to 1 when $i=j=k,\,$ and zero otherwise.

With this tensor you can write $$ \frac{\partial y_i}{\partial a_j} = T_{ijk}\,y_k - y_i\,y_j $$ where the repeated index $k$ is summed.

Answer 3

This is the partial derivative of the softmax function $y_{k'}$ with respect to its activation $a_k$. Someone on this site has already written an excellent answer that explains the full evaluation of this derivative, just with slightly different notation:

Derivative of Softmax with respect to weights

I'm confused by the delta kk' and i have never seen anything like it.

As others have mentioned, the Kronecker delta function $\delta_{kk'}=1$ when the indices match (i.e. $k=k'$) and is $0$ elsewhere. It's a handy way to "select" only part of the multivariate input when performing derivatives over multiply-connected paths like a neural network.

Think of it almost like an if-else statement for a particular link. Also, it's not the only way you can represent this mathematically - some texts use an indicator function like $I[k=k']$ to represent the same logic.

In our case, the partial derivative of the specific activation $y_{k'}$ (indexed by $k'$) is not just with respect to any activation, but to the specific activation $a_k$ indexed by $k$. That's why a lot of the terms in the partial derivative are set to $0$ when $k \ne k'$, using the Kronecker delta.

Another question is do we consider the summation while taking the derivative, why or why not?

If I understand your question correctly - we do consider the summation in the denominator, $\sum_{k'=1}^K e^{a_k{k'}}$. The derivation I linked to above shows how that is done.

https://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function is a bit relevant, but the result of differentiation is different.

It is indeed related, but focusses on a different partial derivative. If you had a Loss function $L$ that is a function of your softmax output $y_{k'}$, then you could go one step further and evaluate this using the chain rule

$$\frac{\partial L}{\partial a_k} = \frac{\partial L}{\partial y_{k'}} \frac{\partial y_{k'}}{\partial a_k}$$

The last term in the above, $\frac{\partial y_{k'}}{\partial a_k}$, is what you are focussing on in your question, while the question you linked to is trying to evaluate $\frac{\partial L}{\partial a_k}$. So they are related but focussed on different terms.

Answer 4

First thing to remember is that when you differentiate an ($n,1$) vector $y$ with respect to a ($n,1$) vector $a$, you get an ($n,n$) matrix, who's first column is the the differentiation w.r.t. $a_1$, 2nd w.r.t $a_2$ ... etc. all the way to $a_n$.

\begin{pmatrix} \frac{\partial y_1}{\partial a_1} & \frac{\partial y_1}{\partial a_2} & \frac{\partial y_1}{\partial a_3} & \cdots & \frac{\partial y_1}{\partial a_n} \\ \frac{\partial y_2}{\partial a_1} & \frac{\partial y_2}{\partial a_2} &\frac{\partial y_2}{\partial a_3} & \cdots & \frac{\partial y_2}{\partial a_n} \\ \vdots & \vdots& \vdots & \ddots & \vdots \\ \frac{\partial y_n}{\partial a_1} & \frac{\partial y_n}{\partial a_2} & \frac{\partial y_n}{\partial a_3} & \cdots & \frac{\partial y_n}{\partial a_n} \end{pmatrix}

Now, since $y_i = \frac{e^{a_i}}{\sum_je^{a_j}}$, let's see the general form of it's derivatives: $$\frac{\partial y_i}{\partial a_i} = \frac{e^{a_i} \sum_je^{a_j} - e^{a_i}e^{a_i}}{(\sum_je^{a_j})^2} = \frac{e^{a_i}}{\sum_je^{a_j}} \frac{\sum_je^{a_j} - e^{a_i}}{\sum_je^{a_j}} = y_i(1-y_i) = y_i - y_i y_i\\ \frac{\partial y_i}{\partial a_k} = \frac{- e^{a_i}e^{a_k}}{(\sum_je^{a_j})^2} = -\frac{e^{a_i}}{\sum_je^{a_j}}\frac{e^{a_k}}{\sum_je^{a_j}} = -y_i y_k $$

So, you can separate the matrix into 2 matrices: $$ \left ( \begin{matrix} y_1 & 0 & 0 & \cdots & 0 \\ 0 & y_2 & 0 & \cdots & 0 \\ \vdots & \vdots& \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & y_n \end{matrix} \right) - \left ( \begin{matrix} y_1 y_1 & y_1 y_2 & y_1 y_3 & \cdots & y_1 y_n \\ y_2 y_1 & y_2 y_2 & y_2 y_3 & \cdots & y_2 y_n \\ \vdots & \vdots& \vdots & \ddots & \vdots \\ y_n y_1 & y_n y_2 & y_n y_3 & \cdots & y_n y_n \end{matrix} \right) $$

You can see that the first matrix correspond to the first term in your equation $y_k \delta_{k k'}$ and the end matrix correspond to the 2nd term $y_k y_{k'}$.

You can check this blog post for more information including graphs and code.