Different definitions of cross entropy loss function not equivalent?

Question

In this question: Different definitions of the cross entropy loss function, two different definitions of cross-entropy cost function are proposed:

$$ C = -\frac{1}{n} \sum_x \sum_j(y_j \ln a_{j}^{L})$$ and $$C = -\frac{1}{n} \sum_x \sum_j (y_j \ln a_{j}^{L} + (1-y_j) \ln(1-a_{j}^{L})).$$

The analysis in the answer to the question I referred to shows that for binary classification (j=2), given that $ \sum_j a_j = 1$ and $ y $ is a one-hot vector, it holds that: $$ C = -\frac{1}{n} \sum_x \sum_{j=1}^2 (y_j \ln a_j) = -\frac{1}{n}\sum_x y_1\ln a_1 + y_2 \ln a_2 = \\ -\frac{1}{n} \sum_x y_1 \ln a_1 + (1 - y_1) \ln (1 - a_1).$$

However, I don't see how this analysis shows that the 2 definitions are equivalent given the assumptions, because in the second definition, if we take $j=2$, it yields: $$C = -\frac{1}{n} \sum_x [y_1 \ln a_1^L + (1 - y_1) \ln (1 - a_1^L) + y_2 \ln a_2^L + (1 - y_2) \ln (1 - a_2^L)].$$

Moreover, I would like to show that the definitions are equivalent for any number of output neurons, not just for 2 neurons.

Answer 1

Your extension of the two-class definition requires a bit of care. If there are more than two classes, we have to stipulate that in this case, $\sum a_i=1$ and that $a_i\ge 0\forall i,$ i.e. the predicted class memberships are all positive and sum to 1. Then the one-hot encoding provides that precisely one of the $y_i=1$ and the rest are $0$.

Then we can write the multi-class cross-entropy as $$C=-\frac{1}{n}\sum_x\sum_i y_i \ln (a_i)$$

Note that the one-hot encoding scheme makes the product 0 for all but one of the $a_i$. Contrast this to the expression you have, in which each predicted value appears twice because it is evaluated once for each of two $y$ vectors, and each $y_i$ has precisely one nonzero value under one-hot encoding. The result is that your expression produces a result which is $2C$ the expression here (because of the restrictions on $a_i$). This can be demonstrated directly using any configuration of $y_i, a_i$ which satisfy our requirements.

There is basically no consequence, since optima will occur for the same parameter values, but I recommend using the definition with is consistent with standard practice.