Deriving the gradients for Softmax logistic regression classifier

Question

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$

In this simple neural network, I am trying to derive the Jacobian for $\frac{\partial{L}}{\partial{z}}$, which is equal to $\frac{\partial{L}}{\partial{z}}\frac{\partial{z}}{\partial{a}}$. This works out to a $1 \times k$ vector.

I believe that the Jacobian for $\frac{\partial{L}}{\partial{z}} = [\frac{\partial{L}}{\partial{z_1}}, \frac{\partial{L}}{\partial{z_2}},...\frac{\partial{L}}{\partial{z_k}}]=[-\frac{y_1}{z_1}, -\frac{y_2}{z_2}, ... , -\frac{y_k}{z_k}]$

$\frac{\partial{z}}{\partial{a}}$ is a Jacobian matrix of $k \times k$ dimensions and $\frac{\partial{z_i}}{\partial{a_j}} = z_i - y_i$ if $i = j$ and $-z_jz_i$ otherwise.

Multiplying $\frac{\partial{L}}{\partial{z}}\frac{\partial{z}}{\partial{a}}$ should give me a $1\times k$ vector with each entry being $\frac{\partial{L}}{\partial{a_k}}$.

I know that $\frac{\partial{L}}{\partial{a_i}}$ somehow be equal to $z_i - y_i$ based on this post. However, I have trouble trying to get that answer.

For the first entry $\frac{\partial{L}}{\partial{a_1}}$, multiplying the rows of $\frac{\partial{L}}{\partial{z}}$ by the columns of $\frac{\partial{z}}{\partial{a}}$ I get

$$ \begin{aligned} \frac{\partial{L}}{\partial{a_1}} &= \frac{\partial{L}}{\partial{z_1}}\frac{\partial{z_1}}{\partial{a_1}} + \frac{\partial{L}}{\partial{z_2}}\frac{\partial{z_2}}{\partial{a_1}} + ... + \frac{\partial{L}}{\partial{z_k}}\frac{\partial{z_2}}{\partial{a_1}} \\[1em] &= -\frac{y_1}{z_1}(z_1 - y_1) - \frac{y_2}{z_2}(-z_2z_1) - \frac{y_3}{z_3}(-z_3z_1) - ... - \frac{y_k}{z_k}(-z_kz_1) \\[1ex] &= -y_1 + \frac{y_1^2}{z_1} + y_2z_1 + y_3z_1 + ... + y_kz_1 \end{aligned} $$ I tried to factor out $z_1$, but I couldn't see any pattern there.

Not sure If I have derived some equations wrongly ? Would appreciate some pointers !

Answer 1

Let's first correct some of the typos:

• You're looking for $\frac{\partial L}{\partial a_1}$ in the end (not $\frac{\partial L}{\partial z_1}$ because you already have it)
• $\frac{\partial z_i}{\partial a_i}$ can't be equal to $z_i-y_i$ because $y_i$ is the label and has nothing to do with the internal variables' derivatives. It is actually $z_i(1-z_i)$.

If we substitute for it $$\begin{aligned} \frac{\partial{L}}{\partial{a_1}} &= \frac{\partial{L}}{\partial{z_1}}\frac{\partial{z_1}}{\partial{a_1}} + \frac{\partial{L}}{\partial{z_2}}\frac{\partial{z_2}}{\partial{a_1}} + ... + \frac{\partial{L}}{\partial{z_k}}\frac{\partial{z_2}}{\partial{a_1}} \\[1em] &= -\frac{y_1}{z_1}z_1(1-z_1) - \frac{y_2}{z_2}(-z_2z_1) - \frac{y_3}{z_3}(-z_3z_1) - ... - \frac{y_k}{z_k}(-z_kz_1) \\[1ex] &= -y_1+z_1\underbrace{(y_1+y_2+...+y_k)}_1=z_1-y_1 \end{aligned}$$ Assuming we have one of the labels as $1$, the sum of labels is $1$.