Need help understanding inverting softmax from Michael Nielsen's book

Question

I have a hard time understanding a softmax problem from the book:

Inverting the softmax layer Suppose we have a neural network with a softmax output layer and the > activations $a^L_j$ are known. Show that the corresponding weighted inputs have the form $z^L_j = \ln a^L_j + C$ for some constant $C$ that is independent of $j$.

Here is how I've approached: $a^L_j= \exp(z^L_j) / \sum(\exp(z^L_k))$. Take the log of both sides, $\ln(a^L_j) = z^L_j - \ln(\sum(\exp(z^L_k)))$. Then, $z^L_j = \ln(a^L_j) + \ln(\sum(\exp(z^L_k)))$ The problem happens here. Why we're allowed to substitute $\ln(\sum(\exp(z^L_k)))$ for $C$ when it has $z^L_j$ in it? Everyone from my research says $C$ is independent of $j$ so it can be $C$. But, doesn't that mean we have to extract $e^L_j$ out of $\ln(\sum(\exp(z^L_k)))$?

Can you please give me a insight into this problem?

Answer 1

Regardless of which $j$ you’re currently considering, the log-sum-exp term you’ve written out (the log–normalizing constant) is the same. That’s the independence that this textbook problem asks about. Rather than pulling the $j$ out, show that it doesn’t matter which $j$ you put in.