Relationship between log-likelihood function and entropy (instead of cross-entropy)

Question

Negative log-likelihood function $$ -\ln f(X\mid\Theta)=-\int_{-\infty}^{\infty}\ln f(x\mid\Theta) \, \mathrm{d}x$$

Differential entropy

$$ h(X) = -\int_{-\infty}^{\infty} f(x) \ln f(x) \, \mathrm{d}x$$

There are alot of comparisons of the negative log-likelihood function of a pdf with its cross-entropy (KL-divergence), since maximizing the likelihood is the same as minimizing cross-entropy, but, given that the log-likelihood is a go-to method for estimating the parameters $\Theta$ of a distribution, $f(x)$, why would one go the next step by additionally tacking on the pdf immediately in front of the log-likelihood function which is all that the entropy formula seems to do (see above).

Some sources even give the equation of log-likelihood to be the entropy formula, except with one of the entries instead being predictions, $\hat{f}(x)$, so I am confused. Also not sure whether positive or negative log-likelihood is more comparable to entropy.

Answer 1

Likelihood and entropy measure different things. The former tells us how likely it is to observe the dataset $X$ given the parameters $\theta$. The latter is supposed to tell us how much uncertainty is in the distribution. So, comparing these two metrics is like comparing weight and hair length of a student: can be done, but somewhat pointless.

Let us compare them anyways. The similarity of equations that you mentioned - additionally tacking on the pdf immediately in front of the log-likelihood function which is all that the entropy formula seems to do - is rather superficial. The equation that you wrote for likelihood is unusual, in that it gives an impression that you're integrating over the domain of the distribution $f(.)$, but that's not how it works in estimation of parameters $\theta$ to which you referred. So, the more common equation is as follows: $$\mathcal L=\sum_if(x_i|\theta)$$ You see how the summation is not over the domain of $f$ but over the observations $x_i\in X$ in the dataset.

Compare this to a Shannon entropy equation: $$h=E[-\log p]=-\sum_ip_i\ln p_i$$ I purposefully changed notation to emphasize that unlike the likelihood function the summation here is over the domain of the probability distribution. In other words, the index $i$ denotes all possible outcomes of the variable, not the observations from the dataset. This alone should immediately show you that Shannon entropy is not a mere enhancement of the likelihood, as it may seem from the way you wrote the equations, that this thing must be measuring something completely different.

The reason why I switched to discrete case from the continuous is mainly didactic: to make a clear distinction between domain, e.g. $\mathbb R$, and the dataset $X$. The continuous form that you used for likelihood can be confusing. However, there is another reason: differential entropy doesn't do what it was meant to do. Shannon entropy matches exactly Gibbs entropy, a fundamental concept in statistical mechanics. Shannon cleverly applied it to the information theory filed, which he pretty much created. However, when he attempted to extend the concept to continuous distributions, his expression for differential entropy, that you used here too, turned out to be incorrect. It has many issues, e.g. it is not dimensionless. So, strictly speaking to compare likelihood and the entropy in continuous case, you'd better use a correct formulation of the entropy itself, not the differential entropy.