8
$\begingroup$

"Entropy" roughly captures the degree of "information" in a probability distribution.

For discrete distributions there is a far more exact interpretation: The entropy of a discrete random variable is a lower bound on the expected number of bits required to transfer the result of the random variable.

But for a continuous random variable, there are uncountably infinite number of outcomes, so we cannot even begin to transfer which exact outcome has occurred in a finite string of bits.

What is an equivalent interpretation of entropy for continuous variables?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Do you have any definition of "degree of information" in a probability distribution? $\endgroup$ – kjetil b halvorsen♦ May 21 '18 at 22:08
  • $\begingroup$ @kjetilbhalverson, i don't see where you're going with this? Isn't the question pretty clear? $\endgroup$ – user56834 May 22 '18 at 3:28
  • $\begingroup$ I think good answers are given here and here. $\endgroup$ – COOLSerdash Jun 3 '18 at 8:38
  • $\begingroup$ @COOLSerdash perfect. Could you make an answer linking to those two, and I’ll give you the points. $\endgroup$ – user56834 Jun 3 '18 at 8:43
  • $\begingroup$ @Programmer2134 I really appreciate it but I don't feel comfortable just posting links without much context (which is discouraged here) and getting points for it. I'm sorry. $\endgroup$ – COOLSerdash Jun 3 '18 at 19:51
5
+25
$\begingroup$

Because of Limiting density of discrete points, the interpretation of $$S = -\sum_x p(x)\ln p(x)$$ cannot be generalized to $$S= -\int dx (p(x)\ln p(x))$$

Because the direct generalization leads to $$S= -\int dx p(x)\ln (p(x)dx) = -\int dx p(x)\ln (p(x)) -\int dx p(x)\ln (dx) $$ Clearly, $\ln dx$ explodes.

Intuitively, since $p(x)dx = 0$, so the reasoning of using fewer bits for encoding something that is more likely to happen does not hold. So, we need to find another way to interpret $S= -\int dx p(x)\ln (p(x)dx)$, and the choice is $KL$ divergence.

Say we have a uniform distribution $q(x)$ in the same state space, then we have $$KL(p(x)\Vert q(x)) = \int dx p(x) \ln (\frac{p(x)dx}{q(x)dx})$$ Since $q(x)$ is just a constant, so we effectively keep the form of $S= -\int dx (p(x)\ln (p(x)dx))$, and at the same time construct a well-defined quantity for the continuous distribution $p(x)$.

So from $KL$ divergence, the entropy of a continuous distribution $p(x)$ can be interpreted as:

If we use a uniform distribution for encoding $p(x)$, then how many bits that is unnecessary on average.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Your last sentence comes to the topic of the question, but it doesn’t actualy answer it: What is the interpretation of this “intrinsic property” then, if it is not the number of bits? $\endgroup$ – user56834 Jun 3 '18 at 7:41
  • 2
    $\begingroup$ The entropy is the expectation of $\ln (1/P)$. It's a matter of mathematical education that people usually prefer to write the definiion as you do, to write first $-\ln P$ and then to take the minus sign outside the integral, but your answer needs that correction. . $\endgroup$ – Nick Cox Jun 3 '18 at 7:47
  • $\begingroup$ @ Nick Cox Thanks for pointing this out, I've edited that. $\endgroup$ – meTchaikovsky Jun 3 '18 at 8:20
  • $\begingroup$ @Programmer2134 I've edited my answer, I hope it addresses the question better. $\endgroup$ – meTchaikovsky Jun 3 '18 at 8:24
  • $\begingroup$ @Programmer2134 Thanks to your question, I found that I just totally misunderstood the interpretation of $-\int p(x) \ln p(x)$. I 've corrected my answer. $\endgroup$ – meTchaikovsky Jun 3 '18 at 9:33
2
$\begingroup$

You discretize the problem via a probability density. A continous random variable has a density $f(x)$, which locally approximates the probably $P(X\in [x,x+\delta x]) \approx f(x)\delta x$, which is now an analogue of the discrete case. And by the theory of calculus, your sums equivalently become integrals over you state space.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I may be missing something, but my question was about an interpretation. I know that an integral is a limit of sums. $\endgroup$ – user56834 May 21 '18 at 18:04
  • 1
    $\begingroup$ This is a very optimistic answer! I believe itis much more complicated $\endgroup$ – kjetil b halvorsen♦ May 21 '18 at 22:06
  • $\begingroup$ @kjetilbhalvorsen: Yea, there are a lot of details shoved under-the-table here. For OP's benefit, see section 2.3.1 of this: crmarsh.com/static/pdf/Charles_Marsh_Continuous_Entropy.pdf $\endgroup$ – Alex R. May 21 '18 at 23:13
  • 1
    $\begingroup$ Could you elaborate this answer? It seems to suggest that you can approximate continuous entropy by discretizing the distribution with small bins. But your link shows this doesn't work, and even says "the formula for continuous entropy is not a derivation of anything" $\endgroup$ – Jonny Lomond May 30 '18 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.