40
$\begingroup$

I am reading about entropy and am having a hard time conceptualizing what it means in the continuous case. The wiki page states the following:

The probability distribution of the events, coupled with the information amount of every event, forms a random variable whose expected value is the average amount of information, or entropy, generated by this distribution.

So if I calculate the entropy associated with a probability distribution that is continuous, what is that really telling me? They give an example about flipping coins, so the discrete case, but if there is an intuitive way to explain through an example like that in the continuous case, that would be great!

If it helps, the definition of entropy for a continuous random variable $X$ is the following:

$$H(X)=-\int P(x)\log_b P(x)dx$$ where $P(x)$ is a probability distribution function.


To try and make this more concrete, consider the case of $X\sim \text{Gamma}(\alpha,\beta)$, then, according to Wikipedia, the entropy is

\begin{align} H(X)&=\mathbb{E}[-\ln(P(X))]\\ &=\mathbb{E}[-\alpha\ln(\beta)+\ln(\Gamma(\alpha))+\ln(\Gamma(\alpha))-(\alpha-1)\ln(X)+\beta X]\\ &=\alpha-\ln(\beta)+\ln(\Gamma(\alpha))+(1-\alpha)\left(\frac{d}{d\alpha}\ln(\Gamma(\alpha))\right) \end{align}

And so now we have calculated the entropy for a continuous distribution (the Gamma distribution) and so if I now evaluate that expression, $H(X)$, given $\alpha$ and $\beta$, what does that quantity actually tell me?

$\endgroup$
9
  • 6
    $\begingroup$ (+1) That quotation references a truly unfortunate passage. It is attempting, in a laborious and opaque way, to describe and interpret the mathematical definition of entropy. That definition is $\int f(x)\log(f(x))dx$. It can be viewed as the expectation of $\log(f(X))$ where $f$ is the pdf of a random variable $X$. It is attempting to characterize $\log(f(x))$ as the "amount of information" associated with the number $x$. $\endgroup$
    – whuber
    Commented Feb 5, 2016 at 19:00
  • 5
    $\begingroup$ It's worth asking, because there is a delicate but important technical issue: the continuous version of entropy does not quite enjoy the same properties as the discrete version (which does have a natural, intuitive intepretation in terms of information). @Tim AFAIK, that thread on Mathematics addresses only the discrete case. $\endgroup$
    – whuber
    Commented Feb 5, 2016 at 19:05
  • 1
    $\begingroup$ @RustyStatistician think of $-\log\left(f\left(x\right)\right)$ as telling you how surprising the outcome x was. You are then calculating expected surprise. $\endgroup$
    – Adrian
    Commented Feb 5, 2016 at 19:06
  • 3
    $\begingroup$ Re the technical issue @whuber references, this may be of interest. $\endgroup$ Commented Feb 5, 2016 at 19:08
  • 3
    $\begingroup$ In case you are interested in technicalities: Entropy is a based off a pseudo-metric called the Kullback-Leibler divergence that is used to describe distances between events in their respective measure, see projecteuclid.org/euclid.aoms/1177729694 for the original (and groudbreaking) paper by Kullback and Leibler. The concept also reappears in model selection criteria like the AIC and BIC. $\endgroup$
    – Jeremias K
    Commented Feb 5, 2016 at 19:21

3 Answers 3

43
+50
$\begingroup$

The entropy tells you how much uncertainty is in the system. Let's say you're looking for a cat, and you know that it's somewhere between your house and the neighbors, which is 1 mile away. Your kids tell you that the probability of a cat being on the distance $x$ from your house is described best by beta distribution $f(x;2,2)$. So a cat could be anywhere between 0 and 1, but more likely to be in the middle, i.e. $x_{max}=1/2$.

enter image description here

Let's plug the beta distribution into your equation, then you get $H=-0.125$.

Next, you ask your wife and she tells you that the best distribution to describe her knowledge of your cat is the uniform distribution. If you plug it to your entropy equation, you get $H=0$.

Both uniform and beta distributions let the cat be anywhere between 0 and 1 miles from your house, but there's more uncertainty in the uniform, because your wife has really no clue where the cat is hiding, while kids have some idea, they think it's more likely to be somewhere in the middle. That's why Beta's entropy is lower than Uniform's.

enter image description here

You might try other distributions, maybe your neighbor tells you the cat likes to be near either of the houses, so his beta distribution is with $\alpha=\beta=1/2$. Its $H$ must be lower than that of uniform again, because you get some idea about where to look for a cat. Guess whether your neighbor's information entropy is higher or lower than your kids'? I'd bet on kids any day on these matters.

enter image description here

UPDATE:

How does this work? One way to think of this is to start with a uniform distribution. If you agree that it's the one with the most uncertainty, then think of disturbing it. Let's look at the discrete case for simplicity. Take $\Delta p$ from one point and add it to another like follows: $$p_i'=p-\Delta p$$ $$p_j'=p+\Delta p$$

Now, let's see how the entropy changes: $$H-H'=p_i\ln p_i-p_i\ln (p_i-\Delta p)+p_j\ln p_j-p_j\ln (p_j+\Delta p)$$ $$=p\ln p-p\ln [p(1-\Delta p/p)]+p\ln p-p\ln [p(1+\Delta p/p)]$$ $$=-\ln (1-\Delta p/p)-\ln (1+\Delta p/p)>0$$ This means that any disturbance from the uniform distribution reduces the entropy (uncertainty). To show the same in continuous case, I'd have to use calculus of variations or something along this line, but you'll get the same kind of result, in principle.

UPDATE 2: The mean of $n$ uniform random variables is a random variable itself, and it's from Bates distribution. From CLT we know that this new random variable's variance shrinks as $n\to\infty$. So, uncertainty of its location must reduce with increase in $n$: we're more and more certain that a cat's in the middle. My next plot and MATLAB code shows how the entropy decreases from 0 for $n=1$ (uniform distribution) to $n=13$. I'm using distributions31 library here.

enter image description here

x = 0:0.01:1;
for k=1:5
    i = 1 + (k-1)*3;
    idx(k) = i;
    f = @(x)bates_pdf(x,i);
    funb=@(x)f(x).*log(f(x));
    fun = @(x)arrayfun(funb,x);
    h(k) = -integral(fun,0,1);
    subplot(1,5+1,k)

    plot(x,arrayfun(f,x))
    title(['Bates(x,' num2str(i) ')'])
    ylim([0 6])
end

subplot(1,5+1,5+1)
plot(idx,h)
title 'Entropy'
$\endgroup$
4
  • 3
    $\begingroup$ (+1) I'll wait to see others interpretations but I really like this one. So it seems like to be able to make use of entropy as a measure of certainty you need to compare it against other distributions? I.e., the number by itself doesn't tell you much? $\endgroup$ Commented Feb 8, 2016 at 17:15
  • 3
    $\begingroup$ @RustyStatistician, I wouldn't say its absolute value is totally meaningless., but yes, it's most useful when used to compare the states of the system. The easy way to internalize entropy is to think of it as measure of uncertainty $\endgroup$
    – Aksakal
    Commented Feb 8, 2016 at 17:17
  • $\begingroup$ Problem with this answer is that the term "uncertainty" is left undefined. $\endgroup$ Commented Nov 10, 2016 at 16:20
  • 2
    $\begingroup$ the term is left uncertain $\endgroup$
    – Aksakal
    Commented Nov 10, 2016 at 18:38
3
$\begingroup$

what does that quantity actually tell me?

I'd like to plug in a straightforward answer as follows:

It's intuitive to illustrate that in a discrete scenario. Suppose that you toss a heavily biased coin, saying the probability of seeing head on each flip is 0.99. Every actual flip tells you very little information because you almost already know that it will be head. But when it comes to a fairer coin, it's harder for you to have any idea what to expect, then every flip tells you more information than any more biased coin. The quantity of information obtained by observing a single toss is equated with $\log \frac{1}{p(x)}$.

What the quantity of the entropy tells us is the information every actual flipping on average(weighted by its probability of occuring) can convey: $E \log \frac{1}{p(x)} = \sum p(x) \log \frac{1}{p(x)} $. The fairer the coin the larger the entropy, and a completely fair coin will be maximally informative.

$\endgroup$
1
$\begingroup$

I don't feel that the most of the answers provided above are answering the question posed, except the comment by whuber. If I understand correctly, the original question pertains to DISCRETE cases as opposed the continuous cases. My impression is that RustyStatistician knows well what entropy means in discrete cases but is not sure of its meaning in continuous cases. Here is my answer: it does not mean much!

The following are my reasons:

  1. I have had the same question for years, have had searched for a satisfactory answer for years, but without success.
  2. To me one of the most important properties of entropy is its label invariance in discrete cases - it does not change its value under permutations on the index set $\{k;k\geq 1\}$, as in $\{p_{k}; k\geq 1\}$. As it is label-invariant, it measures internal volatility of a random element (as opposed to a random variable). In continuous cases (continuous random variables $X$), entropy changes its value if the values on the real line are arbitrarily exchanged. This fact puts entropy in continuous cases in a very different category. It does not mean that it is useless, but is of different meaning and different utility.
  3. I have consulted a world class expert on entropy, his best answer was: if you ask 10 experts the same question, they will give you at least 9 different answers.
  4. Legend has it that Kolmogorov once said to his students that entropy in continuous cases does not mean much. Many years later, I asked one of his students for a reference. He said with heavy Russian accent, "perhaps he just said it."
  5. One day Confucius was asked a question by one of his pupils, "what happens after one dies?" Confucius replied, "it is best to not answer this question until we figured out what happens before one dies."
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.