# Deriving the gradients for Softmax logistic regression classifier

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$

In this simple neural network, I am trying to derive the Jacobian for $$\frac{\partial{L}}{\partial{z}}$$, which is equal to $$\frac{\partial{L}}{\partial{z}}\frac{\partial{z}}{\partial{a}}$$. This works out to a $$1 \times k$$ vector.

I believe that the Jacobian for $$\frac{\partial{L}}{\partial{z}} = [\frac{\partial{L}}{\partial{z_1}}, \frac{\partial{L}}{\partial{z_2}},...\frac{\partial{L}}{\partial{z_k}}]=[-\frac{y_1}{z_1}, -\frac{y_2}{z_2}, ... , -\frac{y_k}{z_k}]$$

$$\frac{\partial{z}}{\partial{a}}$$ is a Jacobian matrix of $$k \times k$$ dimensions and $$\frac{\partial{z_i}}{\partial{a_j}} = z_i - y_i$$ if $$i = j$$ and $$-z_jz_i$$ otherwise.

Multiplying $$\frac{\partial{L}}{\partial{z}}\frac{\partial{z}}{\partial{a}}$$ should give me a $$1\times k$$ vector with each entry being $$\frac{\partial{L}}{\partial{a_k}}$$.

I know that $$\frac{\partial{L}}{\partial{a_i}}$$ somehow be equal to $$z_i - y_i$$ based on this post. However, I have trouble trying to get that answer.

For the first entry $$\frac{\partial{L}}{\partial{a_1}}$$, multiplying the rows of $$\frac{\partial{L}}{\partial{z}}$$ by the columns of $$\frac{\partial{z}}{\partial{a}}$$ I get

\begin{aligned} \frac{\partial{L}}{\partial{a_1}} &= \frac{\partial{L}}{\partial{z_1}}\frac{\partial{z_1}}{\partial{a_1}} + \frac{\partial{L}}{\partial{z_2}}\frac{\partial{z_2}}{\partial{a_1}} + ... + \frac{\partial{L}}{\partial{z_k}}\frac{\partial{z_2}}{\partial{a_1}} \\[1em] &= -\frac{y_1}{z_1}(z_1 - y_1) - \frac{y_2}{z_2}(-z_2z_1) - \frac{y_3}{z_3}(-z_3z_1) - ... - \frac{y_k}{z_k}(-z_kz_1) \\[1ex] &= -y_1 + \frac{y_1^2}{z_1} + y_2z_1 + y_3z_1 + ... + y_kz_1 \end{aligned} I tried to factor out $$z_1$$, but I couldn't see any pattern there.

Not sure If I have derived some equations wrongly ? Would appreciate some pointers !

• You're looking for $$\frac{\partial L}{\partial a_1}$$ in the end (not $$\frac{\partial L}{\partial z_1}$$ because you already have it)
• $$\frac{\partial z_i}{\partial a_i}$$ can't be equal to $$z_i-y_i$$ because $$y_i$$ is the label and has nothing to do with the internal variables' derivatives. It is actually $$z_i(1-z_i)$$.
If we substitute for it \begin{aligned} \frac{\partial{L}}{\partial{a_1}} &= \frac{\partial{L}}{\partial{z_1}}\frac{\partial{z_1}}{\partial{a_1}} + \frac{\partial{L}}{\partial{z_2}}\frac{\partial{z_2}}{\partial{a_1}} + ... + \frac{\partial{L}}{\partial{z_k}}\frac{\partial{z_2}}{\partial{a_1}} \\[1em] &= -\frac{y_1}{z_1}z_1(1-z_1) - \frac{y_2}{z_2}(-z_2z_1) - \frac{y_3}{z_3}(-z_3z_1) - ... - \frac{y_k}{z_k}(-z_kz_1) \\[1ex] &= -y_1+z_1\underbrace{(y_1+y_2+...+y_k)}_1=z_1-y_1 \end{aligned} Assuming we have one of the labels as $$1$$, the sum of labels is $$1$$.