Does log-likelihood cost function in a multinomial classification consider only the output at the neuron that should be active for that class?

Question

Consider a neural network with an output layer of softmax neurons and a log likelihood cost function. For easiness consider one wants to train a MNIST classifier. The output layer will have 9 neurons each one outputting the probability of the corresponding digit.

When training with such a configuration, does the cost function consider only the output at the neuron that should be active for that specific digit?

For example, suppose parameters are updated at every sample (input image) and the next image is a $7$. Let me call $a_j$ the activation of the $j$-th neuron. For this input is the cost function just $C=-\log a_7$ or does $C$ depends also on $a_i, \forall i\in[1,9]$?

Since in softmax layers increasing one probability decreases automatically the others, I expect the former to be correct while the latter to be redundant. Here in eq. 81 and 82 however, it does not seem so. Consider for example the gradient w.r.t. the biases: in the ref it is expressed as:

$$\frac{\partial C}{\partial b_j} = a_j - y_j$$

where $y_j$ is $1$ if $1$ for the seventh neuron (the one that should be active when the image is a $7$), $0$ otherwise. I know the formula is correct, but does the cost function consider just $a_7$?

Answer 1

The negative log-likelihood function is defined as $C = \sum_i y_i log(p_i)$ where $y_i$ is the target for the $i$-th neuron and $p_i$ is the output at that neuron from the softmax function. The targets are usually defined in the training set as one-hot-encoded vecotrs, i.e. the digit 2 corresponds to [0,0,1,0,0,0,0,0,0,0]. So all but one of the $y_i$ are 0 and hence the NLL cost function really ignores those values.

However, as you back propagate that loss to adjust your weights, you have to compute $\frac{\partial C}{\partial b_j^L} = a_j^L - y_j$ and $\frac{\partial C}{\partial w_{jk}^L} = a_k^{L-1}(a_j^L - y_j)$, the derivatives of the cost function with respect to the weights and biases of the layer before the softmax activation function respectively. These of course include each $a_j$, because we are trying to bring the probability at the target neuron as close to 1 as possible, which in term means to decrease the others as well because of the definiton of softmax.

If we were to ignore all but the one for the target, we would:

1. Not implement back propagation correctly
2. Only try increasing values in our network at a certain neuron and not decreasing the rest, which could cause our weights and gradients to explode and generally bad things to happen