# Need help understanding inverting softmax from Michael Nielsen's book

I have a hard time understanding a softmax problem from the book:

Inverting the softmax layer Suppose we have a neural network with a softmax output layer and the > activations $$a^L_j$$ are known. Show that the corresponding weighted inputs have the form $$z^L_j = \ln a^L_j + C$$ for some constant $$C$$ that is independent of $$j$$.

Here is how I've approached: $$a^L_j= \exp(z^L_j) / \sum(\exp(z^L_k))$$. Take the log of both sides, $$\ln(a^L_j) = z^L_j - \ln(\sum(\exp(z^L_k)))$$. Then, $$z^L_j = \ln(a^L_j) + \ln(\sum(\exp(z^L_k)))$$ The problem happens here. Why we're allowed to substitute $$\ln(\sum(\exp(z^L_k)))$$ for $$C$$ when it has $$z^L_j$$ in it? Everyone from my research says $$C$$ is independent of $$j$$ so it can be $$C$$. But, doesn't that mean we have to extract $$e^L_j$$ out of $$\ln(\sum(\exp(z^L_k)))$$?

Can you please give me a insight into this problem?

• Hi, please take a moment to fix your LaTeX. Also, because this is a textbook problem, please tag as self-study. Jul 19, 2021 at 9:51
• Thank you for the modification, Arya. I updated the tag.
– Kay
Jul 19, 2021 at 10:24

Regardless of which $$j$$ you’re currently considering, the log-sum-exp term you’ve written out (the log–normalizing constant) is the same. That’s the independence that this textbook problem asks about. Rather than pulling the $$j$$ out, show that it doesn’t matter which $$j$$ you put in.
• Of course C would change if any of the $z^L_j$s changes. But in terms of the normalizing constant for a _particular_ set of $z$s, the symmetry in the equation makes the $C$ the same for all of them. Jul 19, 2021 at 15:50