6
$\begingroup$

In this question: Different definitions of the cross entropy loss function, two different definitions of cross-entropy cost function are proposed:

$$ C = -\frac{1}{n} \sum_x \sum_j(y_j \ln a_{j}^{L})$$ and $$C = -\frac{1}{n} \sum_x \sum_j (y_j \ln a_{j}^{L} + (1-y_j) \ln(1-a_{j}^{L})).$$

The analysis in the answer to the question I referred to shows that for binary classification (j=2), given that $ \sum_j a_j = 1$ and $ y $ is a one-hot vector, it holds that: $$ C = -\frac{1}{n} \sum_x \sum_{j=1}^2 (y_j \ln a_j) = -\frac{1}{n}\sum_x y_1\ln a_1 + y_2 \ln a_2 = \\ -\frac{1}{n} \sum_x y_1 \ln a_1 + (1 - y_1) \ln (1 - a_1).$$

However, I don't see how this analysis shows that the 2 definitions are equivalent given the assumptions, because in the second definition, if we take $j=2$, it yields: $$C = -\frac{1}{n} \sum_x [y_1 \ln a_1^L + (1 - y_1) \ln (1 - a_1^L) + y_2 \ln a_2^L + (1 - y_2) \ln (1 - a_2^L)].$$

Moreover, I would like to show that the definitions are equivalent for any number of output neurons, not just for 2 neurons.

$\endgroup$
0

1 Answer 1

3
$\begingroup$

Your extension of the two-class definition requires a bit of care. If there are more than two classes, we have to stipulate that in this case, $\sum a_i=1$ and that $a_i\ge 0\forall i,$ i.e. the predicted class memberships are all positive and sum to 1. Then the one-hot encoding provides that precisely one of the $y_i=1$ and the rest are $0$.

Then we can write the multi-class cross-entropy as $$C=-\frac{1}{n}\sum_x\sum_i y_i \ln (a_i)$$

Note that the one-hot encoding scheme makes the product 0 for all but one of the $a_i$. Contrast this to the expression you have, in which each predicted value appears twice because it is evaluated once for each of two $y$ vectors, and each $y_i$ has precisely one nonzero value under one-hot encoding. The result is that your expression produces a result which is $2C$ the expression here (because of the restrictions on $a_i$). This can be demonstrated directly using any configuration of $y_i, a_i$ which satisfy our requirements.

There is basically no consequence, since optima will occur for the same parameter values, but I recommend using the definition with is consistent with standard practice.

$\endgroup$
10
  • $\begingroup$ One important piece here is that a fixed multiplier for the cost function is irrelevant, right? The one definition looks like it will be $j$ times the other definition, which is not equal but is equivalent. $\endgroup$ Aug 8, 2016 at 16:14
  • $\begingroup$ Thank you. I understand that the one-hot encoding makes the product 0 for all but one $a_i $ in the first definition. But if you look at the second definition, this is not the case. There will be many sums in the second definition even with the one-hot encoding scheme. Could you elaborate on this? I must be missing something. $\endgroup$
    – stensootla
    Aug 8, 2016 at 16:23
  • $\begingroup$ @abc The last equation you write is precisely twice the more common definition because it "duplicates" the binary $y$ labels into $y_1, y_2$. They're equivalent up to a scalar multiple (2 exactly). $\endgroup$
    – Sycorax
    Aug 8, 2016 at 17:42
  • $\begingroup$ @GeneralAbrial That duplication is what I'm talking about. There are use cases where one result being precisely double (or precisely $j$ times as many) the other result would be reason to declare them not equivalent. Cost functions are special because what matters is the relationship between the costs of two solutions, which is preserved by multiplication by a non-negative scalar. $\endgroup$ Aug 8, 2016 at 18:10
  • $\begingroup$ Do you mean non-zero scalar? Cross-entropy is opposite in sign and scaled by the number of examples compared to binomial deviance, but multiplication by 0 destroys the "relative fitness" property of cost functions w.r.t. alternative parameter tuples. $\endgroup$
    – Sycorax
    Aug 8, 2016 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.