3
$\begingroup$

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$

In this simple neural network, I am trying to derive the Jacobian for $\frac{\partial{L}}{\partial{z}}$, which is equal to $\frac{\partial{L}}{\partial{z}}\frac{\partial{z}}{\partial{a}}$. This works out to a $1 \times k$ vector.

I believe that the Jacobian for $\frac{\partial{L}}{\partial{z}} = [\frac{\partial{L}}{\partial{z_1}}, \frac{\partial{L}}{\partial{z_2}},...\frac{\partial{L}}{\partial{z_k}}]=[-\frac{y_1}{z_1}, -\frac{y_2}{z_2}, ... , -\frac{y_k}{z_k}]$

$\frac{\partial{z}}{\partial{a}}$ is a Jacobian matrix of $k \times k$ dimensions and $\frac{\partial{z_i}}{\partial{a_j}} = z_i - y_i$ if $i = j$ and $-z_jz_i$ otherwise.

Multiplying $\frac{\partial{L}}{\partial{z}}\frac{\partial{z}}{\partial{a}}$ should give me a $1\times k$ vector with each entry being $\frac{\partial{L}}{\partial{a_k}}$.

I know that $\frac{\partial{L}}{\partial{a_i}}$ somehow be equal to $z_i - y_i$ based on this post. However, I have trouble trying to get that answer.

For the first entry $\frac{\partial{L}}{\partial{a_1}}$, multiplying the rows of $\frac{\partial{L}}{\partial{z}}$ by the columns of $\frac{\partial{z}}{\partial{a}}$ I get

$$ \begin{aligned} \frac{\partial{L}}{\partial{a_1}} &= \frac{\partial{L}}{\partial{z_1}}\frac{\partial{z_1}}{\partial{a_1}} + \frac{\partial{L}}{\partial{z_2}}\frac{\partial{z_2}}{\partial{a_1}} + ... + \frac{\partial{L}}{\partial{z_k}}\frac{\partial{z_2}}{\partial{a_1}} \\[1em] &= -\frac{y_1}{z_1}(z_1 - y_1) - \frac{y_2}{z_2}(-z_2z_1) - \frac{y_3}{z_3}(-z_3z_1) - ... - \frac{y_k}{z_k}(-z_kz_1) \\[1ex] &= -y_1 + \frac{y_1^2}{z_1} + y_2z_1 + y_3z_1 + ... + y_kz_1 \end{aligned} $$ I tried to factor out $z_1$, but I couldn't see any pattern there.

Not sure If I have derived some equations wrongly ? Would appreciate some pointers !

$\endgroup$

1 Answer 1

3
$\begingroup$

Let's first correct some of the typos:

  • You're looking for $\frac{\partial L}{\partial a_1}$ in the end (not $\frac{\partial L}{\partial z_1}$ because you already have it)
  • $\frac{\partial z_i}{\partial a_i}$ can't be equal to $z_i-y_i$ because $y_i$ is the label and has nothing to do with the internal variables' derivatives. It is actually $z_i(1-z_i)$.

If we substitute for it $$\begin{aligned} \frac{\partial{L}}{\partial{a_1}} &= \frac{\partial{L}}{\partial{z_1}}\frac{\partial{z_1}}{\partial{a_1}} + \frac{\partial{L}}{\partial{z_2}}\frac{\partial{z_2}}{\partial{a_1}} + ... + \frac{\partial{L}}{\partial{z_k}}\frac{\partial{z_2}}{\partial{a_1}} \\[1em] &= -\frac{y_1}{z_1}z_1(1-z_1) - \frac{y_2}{z_2}(-z_2z_1) - \frac{y_3}{z_3}(-z_3z_1) - ... - \frac{y_k}{z_k}(-z_kz_1) \\[1ex] &= -y_1+z_1\underbrace{(y_1+y_2+...+y_k)}_1=z_1-y_1 \end{aligned}$$ Assuming we have one of the labels as $1$, the sum of labels is $1$.

$\endgroup$
1
  • 1
    $\begingroup$ Oh my.. I missed quite a few things. Thank you for the clarification ! $\endgroup$
    – calveeen
    Feb 19, 2021 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.