Cost function for cross entropy

Question

I'm currently studying from Hinton's neural network course and he just introduced the cost function used with the softmax output function:

\begin{align} C &= -\sum_{j}t_j\log y_j \\[5pt] y_j &= \frac{e^{z_j}}{\sum_{k \in group}e^{z_k}} \end{align}

In the slides, he says that "C has a very big gradient when the target value is 1 and the output is almost zero". This makes sense as we will want to minimize $C$ since the target and output are not the same. However, what if the target value is 0? Here, wouldn't $C$ always be zero so it doesn't matter what weights you have in $y_j$?

Answer 1

The equation you wrote for $C$ looks like the cross entropy loss for a single data point in a multiclass classification problem. In that case, $y_j$ would be the classifier's estimated probability that the class is $j$ given the input. $t_j$ would be an 'indicator variable' that takes the value $1$ if the true class of the data point is $j$ and $0$ otherwise. $t_j$ must be $1$ for exactly one value of $j$ (because the data point has a single, definite class). Since the sum is taken over all $j$ (i.e. all classes), there would be no problems with everything being zero.