1
$\begingroup$

Entropy of probability distributions is the weighted average of the log probabilities of each observation of a random variable. Does this mean that every random variable that has a probability distribution function (p.d.f.) can be measured with entropy? Are there types of pdf's that cannot be measured with entropy?

How about random variables that have a cumulative distribution function (c.d.f.) only? Does entropy apply somehow to those with only a cdf?

$\endgroup$
7
  • 2
    $\begingroup$ A random variable with a PDF has a differential entropy. A discrete variable has an entropy. The two aren't the same things (despite sharing many properties and part of their name). $\endgroup$
    – whuber
    Jul 29 '20 at 16:00
  • $\begingroup$ so by default, and what most people refer to as entropy, is actually differential entropy then, since entropy is more often spoken of for random variables? $\endgroup$
    – develarist
    Jul 29 '20 at 16:20
  • 1
    $\begingroup$ I think most people know about Shannon's entropy. Many think that it works on all r.v.s, and that its expansion to continuous r.v.s is trivial, like moments, but it is not. $\endgroup$
    – Aksakal
    Jul 29 '20 at 16:29
  • 1
    $\begingroup$ So, for a random variable with a mixed type distribution, such as a mixture of an (absoultely) continuous, such as a normal or exponential, and some atoms, it is not clear how to define entropy. $\endgroup$ Jul 29 '20 at 18:11
  • 2
    $\begingroup$ With that comment you seem to conflate the very different meanings of multivariate, unimodal, and mixture distribution, making it almost impossible to formulate a reply. As stated before: discrete random variables (univariate or multivariate, it makes no difference) have entropies and continuous random variables (univariate or multivariate) have differential entropies. $\endgroup$
    – whuber
    Jul 29 '20 at 18:41
2
$\begingroup$

If you mean differential entropy for continuous distributions then there are distributions whose entropy is undefined. So, the answer is NO.

For instance, look at the Warning on p.21 and the Footnote 4 in Polyanskiy and Wu, “Lecture notes on information theory”:

Shannon was probably the first who came up with a notion of entropy in information theory. So, he came up with a formula for discrete entropy, then by analogy wrote the one for continuous randoms which is called differential entropy. However, it turned out not to be the literal expansion from discrete to continuous. The latter is a different equation. Hence, I say that there are at least two different entropy equations for continuous variables.

The entropy as concept was introduced long before Shannon in statistical mechanics and thermodynamics.

$\endgroup$
10
  • $\begingroup$ didn't know about differential entropy. is the standard entropy the same as differential entropy? what are all the different types of entropy $\endgroup$
    – develarist
    Jul 29 '20 at 16:17
  • $\begingroup$ @develarist, the definition (shannon's) that you gave in the question cannot be applied to continuous distributions. the probability of any observation is zero. so you have to come up with something different for continuous, I believe there are a couple out there $\endgroup$
    – Aksakal
    Jul 29 '20 at 16:21
  • $\begingroup$ one comment says differential entropy is for random variables while yours said it's for continuous variables. does this mean that random variables are continuous variables and not discrete varaibles? $\endgroup$
    – develarist
    Jul 29 '20 at 16:23
  • 1
    $\begingroup$ random variables can be continuous or discrete. the main difference is that discrete ones have countable set of outcomes, i.e. you can enumerate all outcomes with an positive integer, e.g. Bernoulli or Poisson. continuous r.v.s cannot be enumerated, an example is Gaussian or Beta distributions. you can define some information measure to both types, including an entropy, but the same equation won't work for both discrete and continuous, at least not in every case $\endgroup$
    – Aksakal
    Jul 29 '20 at 16:25
  • $\begingroup$ the document's section on differential entropy you gave in the answer is quite long. can you list some differential entropy measures? $\endgroup$
    – develarist
    Jul 29 '20 at 16:32
1
$\begingroup$

Every random variable has a probability distribution. You can say a pdf is unknown, but I don't think you can say that it doesn't exist.

As long as you have a random variable, you can measure its entropy.

If pdf is unknown, you can estimate pdf. You can also estimate entropy directly.

For example, an estimator for entropy is this:

"Summing the log of the nearest neighbors distances: a simple estimator for the entropy from samples" enter image description here

Update:

Any random variable can be described by its cumulative distribution function, which describes the probability that the random variable will be less than or equal to a certain value [1]. But, there are also some random variables that doesn't have a pdf nor a pmf, for example Cantor distribution. So, you can also say that a pdf doesn't exist.

$\endgroup$
6
  • 3
    $\begingroup$ "PDF" refers to a density. Many random variables have no density functions. Are you perhaps using "PDF" where you meant "CDF"? $\endgroup$
    – whuber
    Jul 29 '20 at 15:59
  • 2
    $\begingroup$ I second whuber's comment. The density doesn't have to exist (Cantor distribution, for instance), but the CDF will exist. $\endgroup$
    – Dave
    Jul 29 '20 at 16:01
  • $\begingroup$ does the blue figure contain an occuring process, where clusters of the original data are gradually being formed? the scatter plots look like they have different numbers of observations... and what is the purple figure showing? i thought entropy is a scalar value $\endgroup$
    – develarist
    Jul 29 '20 at 16:16
  • $\begingroup$ ah... didn't know about this Cantor distribution. Thanks for clarifying this. $\endgroup$
    – moh
    Jul 29 '20 at 17:19
  • $\begingroup$ @develarist. Yeah, I think the blue one is like sampling more and more from the true distribution, which is the purple one. $\endgroup$
    – moh
    Jul 29 '20 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.