2
$\begingroup$

I have a hard time understanding a softmax problem from the book:

Inverting the softmax layer Suppose we have a neural network with a softmax output layer and the > activations $a^L_j$ are known. Show that the corresponding weighted inputs have the form $z^L_j = \ln a^L_j + C$ for some constant $C$ that is independent of $j$.

Here is how I've approached: $a^L_j= \exp(z^L_j) / \sum(\exp(z^L_k))$. Take the log of both sides, $\ln(a^L_j) = z^L_j - \ln(\sum(\exp(z^L_k)))$. Then, $z^L_j = \ln(a^L_j) + \ln(\sum(\exp(z^L_k)))$ The problem happens here. Why we're allowed to substitute $\ln(\sum(\exp(z^L_k)))$ for $C$ when it has $z^L_j$ in it? Everyone from my research says $C$ is independent of $j$ so it can be $C$. But, doesn't that mean we have to extract $e^L_j$ out of $\ln(\sum(\exp(z^L_k)))$?

Can you please give me a insight into this problem?

$\endgroup$
2
  • $\begingroup$ Hi, please take a moment to fix your LaTeX. Also, because this is a textbook problem, please tag as self-study. $\endgroup$ Jul 19, 2021 at 9:51
  • 1
    $\begingroup$ Thank you for the modification, Arya. I updated the tag. $\endgroup$
    – Kay
    Jul 19, 2021 at 10:24

1 Answer 1

3
$\begingroup$

Regardless of which $j$ you’re currently considering, the log-sum-exp term you’ve written out (the log–normalizing constant) is the same. That’s the independence that this textbook problem asks about. Rather than pulling the $j$ out, show that it doesn’t matter which $j$ you put in.

$\endgroup$
3
  • $\begingroup$ Sorry I'm still not with you, why we're not considering the fact that C would increase as z^L_j increases? What makes every variable in the equation fixed values so that we care about only index? $\endgroup$
    – Kay
    Jul 19, 2021 at 11:47
  • $\begingroup$ Of course C would change if any of the $z^L_j$s changes. But in terms of the normalizing constant for a _particular_ set of $z$s, the symmetry in the equation makes the $C$ the same for all of them. $\endgroup$ Jul 19, 2021 at 15:50
  • $\begingroup$ I see. Thank you very much! $\endgroup$
    – Kay
    Jul 19, 2021 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.