6
$\begingroup$

Why do people sometimes write the entropy as a function on a random variable?

For example, in some class notes I am seeing:

$$ \sum_X Q(X) \log Q(X) = - H_Q(X) $$

I realize this is not to be taken as super formal since there is abuse of notation mixing the random variable and its value, but even after taking that into account this doesn't seem to make a lot of sense because the entropy is a function of the distribution $Q$ alone. Why not simply write $H_Q$ or even $H(Q)$? Why keep the $X$ around?

Improve this question
$\endgroup$
2
  • $\begingroup$ Interesting question. Same applies to E[X] and Var[X] too. $\endgroup$ – user2705196 Feb 26 at 19:50
  • $\begingroup$ Good point! And very interesting too, because I find E[X] and Var[X] very natural and intuitive while finding H_Q(X) not so intuitive. I guess when talking about expectation and variance, we are often focused on the random variable and there is a single distribution on that support, whereas entropy is often discussed in the context of more than one distribution over the same support. $\endgroup$ – user118967 Feb 28 at 6:12
9
$\begingroup$

This is notation that bypasses the distribution (similar to moments)

As can be seen from the formula, the entropy is fully determined by the probability distribution of the random variable, not the random variable itself. In the formula you give in your question the object $X$ is used as an index of summation, so the summation does not depend on $X$ (it is "summed out" of the equation). (Note that it is bad notation to use the same notation for the random variable and the index of summation.) The notation you are referring to is similar to the notation for the moments of random variables, which are also fully determined by the distribution of those random variables. I will try to explain the alternative notational methods here.

Two possible notations: If you take $Q$ to denote the mass function of the random variable (and let $\mathscr{X}$ denote its support) then a natural notation for the entropy would be:

$$H(Q) \equiv - \sum_{x \in \mathscr{X}} Q(x) \log Q(x).$$

However, if you accept the notation for moments, you could also reasonably denote the entropy as:

$$H(X) \equiv \mathbb{E}(-\log(X)) = - \sum_{x \in \mathscr{X}} Q(x) \log Q(x).$$

These two alternatives frame the notation either as a function of the distribution, or a function of the random variable. In the latter case we use the same notational convention used for moments of random variables, which are also functions of the underlying distribution of the random variable.

Technical issues with moment notation: From a technical perspective, a random variable is a mapping on the sample space. If we are working in the probability space $(\Omega, \mathscr{G}, \mathbb{P})$ then a random variable is a measureable function $X: \Omega \rightarrow \mathbb{R}$ from the sample space to the real numbers. Importantly, the random variable does not contain the information on its own distribution, so it is impossible to define the entropy of the distribution ---or any moments of the distribution--- purely as a function of the random variable. It is possible to obtain the probability distribution from the random variable $X$ and the probability measure $\mathbb{P}$, so such a function can have both arguments, but it cannot be a function only of the random variable.

This means that whenever we use moment notation (where we express moments as a function of the random variable rather than the distribution), the functions must be implicitly conditional on the underlying probability measure $\mathbb{P}$ or the specific distribution of the random variable in question. Sometimes we have a function where one or more arguments are only implicit, and do not appear in the function notation, and so it is certainly possible to write moments (and the entropy function) in this way. It is in common usage in statistical problems. This notation is a convenience because it allows you to relate the moments/entropy directly to random variables rather than their distributions.

Improve this answer
$\endgroup$
5
  • $\begingroup$ If you used $\Omega$ as a set of all possible outcomes why you also used $\mathscr{X}$ Aren't these the same support of $X$ $\endgroup$ – Easy Points Feb 26 at 15:52
  • $\begingroup$ Also isn't it better to use $H[X]$ instead $H(Q)$? $\endgroup$ – Easy Points Feb 26 at 15:54
  • $\begingroup$ Thanks for the clarification on $\mathscr{X}$, although it is very common to write $H[X]$ we need to change. $\endgroup$ – Easy Points Feb 26 at 22:03
  • $\begingroup$ Update: I have re-thought this answer and made a major change. Apologies to anyone who upvoted prior to my change of views. $\endgroup$ – Ben Mar 5 at 0:29
  • $\begingroup$ You may consider using $H[X]$ to further distinguish the two for the second case. $\endgroup$ – Easy Points Mar 5 at 10:01
4
$\begingroup$

It makes sense if you remember that X is a statistical ensemble, which has associated with it some probability distribution. For instance, the Wikipedia article for information-theoretic entropy consistently uses the notation $H(X)$. In that sense, it’s actually the Q that’s extraneous.

I can speculate two reasons your instructor includes the Q there. The Q could be there to remind you what the associated probability distribution is. It could also be a preface to introducing concepts like cross-entropy, where you may take your average over a different distribution P. (The cross-entropy Wikipedia article exclusively uses distributions over the same support as arguments to the cross-entropy function.)

Improve this answer
$\endgroup$
1
$\begingroup$

$H$ is just a letter some smart guy use for Entropy. It is possible to use even $E$ for entropy but then there will be confusion with the expected value.

In here $X \sim \mathcal D$, means $\mathcal D$ is probability distribution of discrete random variable $X$.

$$ H[X] = - \sum_{x \sim \mathcal D, \\ x \in R_X} P(x) \log_2 P(x) $$

where:

  • $P(x)$ is probability that $X=x$
  • $x \sim \mathcal D$ ($x$ is also drawn from $\mathcal D$)
  • $R_X$ is the support of $X$ (the set of all possible values $X$ can take)
Improve this answer
$\endgroup$
7
  • $\begingroup$ The notations here are ambiguous/confusing. $H[X]$ suggests that $X$ is the object whose entropy you're measuring. But then you also use $X$ as an index in the sum; it can't be both simultaneously. Also, some of the options here use generic symbols like $Pr$ to denote probability, without specifying the distribution $\endgroup$ – user20160 Feb 26 at 12:37
  • $\begingroup$ @user20160 is it bad still? $\endgroup$ – Easy Points Feb 26 at 14:36
  • 1
    $\begingroup$ Your summation notation is nonstandard and potentially confusing. Because you are using a sum, you must be assuming $X$ is discrete. Thus, the sum is a sum over all real numbers; but since its terms are zero except for the values of $X$ with positive probability, you may also write it as a sum over the support of $X.$ As usual, we are conventionally supposing "$P(x)\log_2 P(x)$" means $0$ when $P(x)=0.$ $\endgroup$ – whuber♦ Feb 26 at 14:58
  • $\begingroup$ Yes, $X$ is discrete RV, this is what I assumed since OP used the $\sum$ as well. How I can denote that $x$ is a sampled from $X$, usually there is an index $i$ from 1 to $N$ means $N$ samples. I will check what is support. Good hints. $\endgroup$ – Easy Points Feb 26 at 15:14
  • $\begingroup$ @whuber, I took your additional inputs and applied to the entropy formula for DRV. $\endgroup$ – Easy Points Feb 26 at 15:22
1
$\begingroup$

A possible perspective is that it better accommodates the notation for conditional entropy (or mutual information): consider the common notation $H_P(X|Y)$, where $P_{XY}$ is understood to refer to the joint distribution on $XY$, versus alternatives one might consider such as e.g. $H(P_X|P_Y)$, which might mislead one into thinking it is a function of the marginal distributions $P_X$ and $P_Y$ alone.

(Of course, $H(P_X|P_Y)$ is not the only way one could try to denote this, but it seems very likely that any notation for conditional entropy will need to refer to $X$, $Y$ and the joint distribution $P_{XY}$ in some form - so you might as well put it together as $H_P(X|Y)$.)

Though in any case, you're not alone in suggesting the notation $H(P)$ when there is only one random variable - it's used in some places, but mostly when there is no danger of confusion along the lines of what's mentioned above.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.