Different definitions of cross entropy loss function not equivalent?

In this question: Different definitions of the cross entropy loss function, two different definitions of cross-entropy cost function are proposed:

$$C = -\frac{1}{n} \sum_x \sum_j(y_j \ln a_{j}^{L})$$ and $$C = -\frac{1}{n} \sum_x \sum_j (y_j \ln a_{j}^{L} + (1-y_j) \ln(1-a_{j}^{L})).$$

The analysis in the answer to the question I referred to shows that for binary classification (j=2), given that $\sum_j a_j = 1$ and $y$ is a one-hot vector, it holds that: $$C = -\frac{1}{n} \sum_x \sum_{j=1}^2 (y_j \ln a_j) = -\frac{1}{n}\sum_x y_1\ln a_1 + y_2 \ln a_2 = \\ -\frac{1}{n} \sum_x y_1 \ln a_1 + (1 - y_1) \ln (1 - a_1).$$

However, I don't see how this analysis shows that the 2 definitions are equivalent given the assumptions, because in the second definition, if we take $j=2$, it yields: $$C = -\frac{1}{n} \sum_x [y_1 \ln a_1^L + (1 - y_1) \ln (1 - a_1^L) + y_2 \ln a_2^L + (1 - y_2) \ln (1 - a_2^L)].$$

Moreover, I would like to show that the definitions are equivalent for any number of output neurons, not just for 2 neurons.

• Thank you, but I have already linked to the question you provided in my post :) – stensootla Aug 9 '16 at 8:49
• oops.. my bad comment deleted :) – dontloo Aug 9 '16 at 8:57

1 Answer

Your extension of the two-class definition requires a bit of care. If there are more than two classes, we have to stipulate that in this case, $\sum a_i=1$ and that $a_i\ge 0\forall i,$ i.e. the predicted class memberships are all positive and sum to 1. Then the one-hot encoding provides that precisely one of the $y_i=1$ and the rest are $0$.

Then we can write the multi-class cross-entropy as $$C=-\frac{1}{n}\sum_x\sum_i y_i \ln (a_i)$$

Note that the one-hot encoding scheme makes the product 0 for all but one of the $a_i$. Contrast this to the expression you have, in which each predicted value appears twice because it is evaluated once for each of two $y$ vectors, and each $y_i$ has precisely one nonzero value under one-hot encoding. The result is that your expression produces a result which is $2C$ the expression here (because of the restrictions on $a_i$). This can be demonstrated directly using any configuration of $y_i, a_i$ which satisfy our requirements.

There is basically no consequence, since optima will occur for the same parameter values, but I recommend using the definition with is consistent with standard practice.

• One important piece here is that a fixed multiplier for the cost function is irrelevant, right? The one definition looks like it will be $j$ times the other definition, which is not equal but is equivalent. – Matthew Graves Aug 8 '16 at 16:14
• Thank you. I understand that the one-hot encoding makes the product 0 for all but one $a_i$ in the first definition. But if you look at the second definition, this is not the case. There will be many sums in the second definition even with the one-hot encoding scheme. Could you elaborate on this? I must be missing something. – stensootla Aug 8 '16 at 16:23
• @abc The last equation you write is precisely twice the more common definition because it "duplicates" the binary $y$ labels into $y_1, y_2$. They're equivalent up to a scalar multiple (2 exactly). – Sycorax Aug 8 '16 at 17:42
• @GeneralAbrial That duplication is what I'm talking about. There are use cases where one result being precisely double (or precisely $j$ times as many) the other result would be reason to declare them not equivalent. Cost functions are special because what matters is the relationship between the costs of two solutions, which is preserved by multiplication by a non-negative scalar. – Matthew Graves Aug 8 '16 at 18:10
• Do you mean non-zero scalar? Cross-entropy is opposite in sign and scaled by the number of examples compared to binomial deviance, but multiplication by 0 destroys the "relative fitness" property of cost functions w.r.t. alternative parameter tuples. – Sycorax Aug 8 '16 at 18:14