4
$\begingroup$

When we compute the cross-entropy within the machine learning context, we use the following formula:

$$ CE(t, p) = -\sum_{i=1}^{N} t_i \ \log(p_i) $$

Where $t$ is the target class probability, and $p$ is the predicted class probability.

My question is, this metric obviously gives us different loss values if the $t$ and $p$ were reversed in the above formula. So, what is the rationale, and reason(s), for the the targets $t$ being outside the log, and the predictions $p$ being inside the log? Why this configuration, and not the opposite?

$\endgroup$
3
  • $\begingroup$ There should be a negative sign in the expression for cross entropy. Couldn't edit it in because system complained it was too few characters. $\endgroup$
    – user20160
    Jun 10, 2016 at 19:12
  • $\begingroup$ @user20160 Don't know what's up with not letting you make a one character edit, but it let me. So the minus sign is now in place. $\endgroup$ Jun 10, 2016 at 19:29
  • 1
    $\begingroup$ The asymmetry is because one is the true probability with which you take expectations (ie outside log) [think expected error] and the other is the predicted probability. $\endgroup$
    – seanv507
    Jun 10, 2016 at 20:09

1 Answer 1

4
$\begingroup$

I'll rewrite the cross entropy in terms of distributions $q$ and $z$, so we can save your notation $t$ and $p$ for explicitly talking about classifiers. The cross entropy is:

$$H(q, z) = -\sum_{x} q(x) \log z(x)$$

The information theoretic meaning of the cross entropy is this: Say $q$ is a distribution that generates some data. We want to encode the data using a set of symbols (e.g. to transmit it over a channel, or store it). We'd prefer to use short codes, because they require fewer resources to transmit/store. There's a fundamental relationship between code length and entropy. It turns out that, for the optimal code, the average number of bits per symbol is given by the entropy of the distribution. No encoding can be shorter than this without destroying information. The intuition is that we should assign short codes to high probability events (because they'll occur more frequently) and longer codes to low probability events. Looking at the definition of entropy:

$$H(q) = -\sum_x q(x) \log q(x)$$

This is the expected value of $-\log q(x)$, which is the optimal code length for event $x$ (given in bits if the log is base 2, or nats if it's the natural log).

But, say we don't know $q$, and instead we have some 'proxy' distribution $z$. We can design an optimal code for $z$, even though the data are generated by $q$. In that case, how many bits/nats per symbol will we use on average to encode the data? The answer to this question is given by the cross entropy $H(q, z)$. Looking at the definition of the cross entropy, we can see that it's the expected value of $-\log z(x)$ with respect to distribution $q$. Here, $-\log z(x)$ is the optimal code length for event $x$, using the code based on $z$. The expectation is taken over $q$ because that's what generated the data. As our proxy distribution $z$ becomes closer to the data-generating distribution $q$, our code will be more efficient, and the cross entropy will be lower. It's minimum possible value is $H(q)$, when $z$ = $q$.

Bringing things back to classification, we have $t$ as the true/observed distribution of class labels, and $p$ as the classifier's estimated distribution over class labels. Plugging these concepts into the description of cross entropy, it makes sense to use $H(t, p)$ because $t$ is the data-generating distribution and $p$ is the 'proxy distribution'.

Here's another argument. When using hard class labels, people treat distribution $t$ as the empirical distribution, which assigns probability $1$ to the true/observed class label $i$, and $0$ to all others. In that case, the cross entropy reduces to: $-\log p(i)$. Summing over all data points, this is just the negative log likelihood. In that case, minimizing the cross entropy loss is equivalent to maximum likelihood estimation.

$\endgroup$
4
  • $\begingroup$ Thanks this was quite eye opening and helpful. One thing I think I am still not quite getting, is the part on where we want to design an optimal code for 'z', without knowing 'q'. Im not quite sure how exactly to interpret this. If we do not know q, how can we compute anything relative to it? Thanks! $\endgroup$
    – Creatron
    Jun 10, 2016 at 20:58
  • $\begingroup$ Yes, calculating cross entropy in closed form requires an expression for both $q$ and $z$. On one hand, you could interpret this as a hypothetical (e.g. "suppose we acted like we didn't know q, what would happen?"). On the other hand, if we truly don't know $q$ but have access to data generated by it, the expected value in the cross entropy can be estimated by averaging over the data. This is typically the situation with fitting models (if we knew the underlying distribution, we'd have no reason to fit a model in the first place). $\endgroup$
    – user20160
    Jun 10, 2016 at 21:26
  • $\begingroup$ Good answer. I agree that this direction of CE is more intuitive. However, we can think of minimizing H(q,z) as the same as minimizing KL(q||z). While KL is not symmetric, both KL(q||z) and KL(z||q) can be seen as similarity measures for q and z. From that reasoning, both H(q,z) and H(z,q) would be acceptable loss functions for approximating q with z. Does that hold? $\endgroup$
    – user118967
    Dec 2, 2020 at 13:04
  • $\begingroup$ I've posted the above comment as a full question at stats.stackexchange.com/questions/498997/… $\endgroup$
    – user118967
    Dec 2, 2020 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.