32
$\begingroup$

I am reading about entropy and am having a hard time conceptualizing what it means in the continuous case. The wiki page states the following:

The probability distribution of the events, coupled with the information amount of every event, forms a random variable whose expected value is the average amount of information, or entropy, generated by this distribution.

So if I calculate the entropy associated with a probability distribution that is continuous, what is that really telling me? They give an example about flipping coins, so the discrete case, but if there is an intuitive way to explain through an example like that in the continuous case, that would be great!

If it helps, the definition of entropy for a continuous random variable $X$ is the following:

$$H(X)=-\int P(x)\log_b P(x)dx$$ where $P(x)$ is a probability distribution function.


To try and make this more concrete, consider the case of $X\sim \text{Gamma}(\alpha,\beta)$, then, according to Wikipedia, the entropy is

\begin{align} H(X)&=\mathbb{E}[-\ln(P(X))]\\ &=\mathbb{E}[-\alpha\ln(\beta)+\ln(\Gamma(\alpha))+\ln(\Gamma(\alpha))-(\alpha-1)\ln(X)+\beta X]\\ &=\alpha-\ln(\beta)+\ln(\Gamma(\alpha))+(1-\alpha)\left(\frac{d}{d\alpha}\ln(\Gamma(\alpha))\right) \end{align}

And so now we have calculated the entropy for a continuous distribution (the Gamma distribution) and so if I now evaluate that expression, $H(X)$, given $\alpha$ and $\beta$, what does that quantity actually tell me?

$\endgroup$
  • 5
    $\begingroup$ (+1) That quotation references a truly unfortunate passage. It is attempting, in a laborious and opaque way, to describe and interpret the mathematical definition of entropy. That definition is $\int f(x)\log(f(x))dx$. It can be viewed as the expectation of $\log(f(X))$ where $f$ is the pdf of a random variable $X$. It is attempting to characterize $\log(f(x))$ as the "amount of information" associated with the number $x$. $\endgroup$ – whuber Feb 5 '16 at 19:00
  • 5
    $\begingroup$ It's worth asking, because there is a delicate but important technical issue: the continuous version of entropy does not quite enjoy the same properties as the discrete version (which does have a natural, intuitive intepretation in terms of information). @Tim AFAIK, that thread on Mathematics addresses only the discrete case. $\endgroup$ – whuber Feb 5 '16 at 19:05
  • 1
    $\begingroup$ @RustyStatistician think of $-\log\left(f\left(x\right)\right)$ as telling you how surprising the outcome x was. You are then calculating expected surprise. $\endgroup$ – Adrian Feb 5 '16 at 19:06
  • 3
    $\begingroup$ Re the technical issue @whuber references, this may be of interest. $\endgroup$ – Sean Easter Feb 5 '16 at 19:08
  • 3
    $\begingroup$ In case you are interested in technicalities: Entropy is a based off a pseudo-metric called the Kullback-Leibler divergence that is used to describe distances between events in their respective measure, see projecteuclid.org/euclid.aoms/1177729694 for the original (and groudbreaking) paper by Kullback and Leibler. The concept also reappears in model selection criteria like the AIC and BIC. $\endgroup$ – Jeremias K Feb 5 '16 at 19:21
31
+50
$\begingroup$

The entropy tells you how much uncertainty is in the system. Let's say you're looking for a cat, and you know that it's somewhere between your house and the neighbors, which is 1 mile away. Your kids tell you that the probability of a cat being on the distance $x$ from your house is described best by beta distribution $f(x;2,2)$. So a cat could be anywhere between 0 and 1, but more likely to be in the middle, i.e. $x_{max}=1/2$.

enter image description here

Let's plug the beta distribution into your equation, then you get $H=-0.125$.

Next, you ask your wife and she tells you that the best distribution to describe her knowledge of your cat is the uniform distribution. If you plug it to your entropy equation, you get $H=0$.

Both uniform and beta distributions let the cat be anywhere between 0 and 1 miles from your house, but there's more uncertainty in the uniform, because your wife has really no clue where the cat is hiding, while kids have some idea, they think it's more likely to be somewhere in the middle. That's why Beta's entropy is lower than Uniform's.

enter image description here

You might try other distributions, maybe your neighbor tells you the cat likes to be near either of the houses, so his beta distribution is with $\alpha=\beta=1/2$. Its $H$ must be lower than that of uniform again, because you get some idea about where to look for a cat. Guess whether your neighbor's information entropy is higher or lower than your kids'? I'd bet on kids any day on these matters.

enter image description here

UPDATE:

How does this work? One way to think of this is to start with a uniform distribution. If you agree that it's the one with the most uncertainty, then think of disturbing it. Let's look at the discrete case for simplicity. Take $\Delta p$ from one point and add it to another like follows: $$p_i'=p-\Delta p$$ $$p_j'=p+\Delta p$$

Now, let's see how the entropy changes: $$H-H'=p_i\ln p_i-p_i\ln (p_i-\Delta p)+p_j\ln p_j-p_j\ln (p_j+\Delta p)$$ $$=p\ln p-p\ln [p(1-\Delta p/p)]+p\ln p-p\ln [p(1+\Delta p/p)]$$ $$=-\ln (1-\Delta p/p)-\ln (1+\Delta p/p)>0$$ This means that any disturbance from the uniform distribution reduces the entropy (uncertainty). To show the same in continuous case, I'd have to use calculus of variations or something along this line, but you'll get the same kind of result, in principle.

UPDATE 2: The mean of $n$ uniform random variables is a random variable itself, and it's from Bates distribution. From CLT we know that this new random variable's variance shrinks as $n\to\infty$. So, uncertainty of its location must reduce with increase in $n$: we're more and more certain that a cat's in the middle. My next plot and MATLAB code shows how the entropy decreases from 0 for $n=1$ (uniform distribution) to $n=13$. I'm using distributions31 library here.

enter image description here

x = 0:0.01:1;
for k=1:5
    i = 1 + (k-1)*3;
    idx(k) = i;
    f = @(x)bates_pdf(x,i);
    funb=@(x)f(x).*log(f(x));
    fun = @(x)arrayfun(funb,x);
    h(k) = -integral(fun,0,1);
    subplot(1,5+1,k)

    plot(x,arrayfun(f,x))
    title(['Bates(x,' num2str(i) ')'])
    ylim([0 6])
end

subplot(1,5+1,5+1)
plot(idx,h)
title 'Entropy'
$\endgroup$
  • 1
    $\begingroup$ (+1) I'll wait to see others interpretations but I really like this one. So it seems like to be able to make use of entropy as a measure of certainty you need to compare it against other distributions? I.e., the number by itself doesn't tell you much? $\endgroup$ – RustyStatistician Feb 8 '16 at 17:15
  • 1
    $\begingroup$ @RustyStatistician, I wouldn't say its absolute value is totally meaningless., but yes, it's most useful when used to compare the states of the system. The easy way to internalize entropy is to think of it as measure of uncertainty $\endgroup$ – Aksakal Feb 8 '16 at 17:17
  • $\begingroup$ Problem with this answer is that the term "uncertainty" is left undefined. $\endgroup$ – kjetil b halvorsen Nov 10 '16 at 16:20
  • 1
    $\begingroup$ the term is left uncertain $\endgroup$ – Aksakal Nov 10 '16 at 18:38
  • $\begingroup$ This is very nice. $\endgroup$ – Astrid May 2 '18 at 9:58
1
$\begingroup$

I'd like to add a straightforward answer to this question:

what does that quantity actually tell me?

It's intuitive to illustrate that in a discrete scenario. Suppose that you toss a heavily biased coin, saying the probability of seeing a head on each flip is 0.99. Every actual flip tells you very little information because you almost already know that it will be head. But when it comes to a fairer coin, it't harder for you to have any idear what to expect, then every flip tells you more information than any more biased coin. The quantity of information obtained by observing a single toss is equated with $\log \frac{1}{p(x)}$.

What the quantity of the entropy tells us is the information every actual flipping on (weighted) average can convey: $E \log \frac{1}{p(x)} = \sum p(x) \log \frac{1}{p(x)} $. The fairer the coin the larger the entropy, and a completely fair coin will be maximally informative.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.