If $X$ is a real random variable with a pdf, variance/standard deviation is a measure of $X$'s dispersion about the pdf's central tendency, which in turn is referred to as the mean of $X$. For many, variance acts as a measure of risk and uncertainty.

Shannon entropy is a measure of the unpredictability of $X$, and is frequently compared with and does have mathematical links to variance. The estimated entropy of $X$'s pdf has been shown to scale alongside variance, in that, as $X$ is transformed by scaling it with increasing variance, entropy will correspondingly shift or grow in value, indicating more spread in the distribution like how the increasing variance indicates more dispersion. Transformations of the mean, however, will not impact the entropy rate.

What is the information analogue of the mean then?

If there is none, does this show a fundamental weakness in information theory? Is entropy and mutual information's inability to distinguish between negative and positive source data for its input probabilities make these measures blind/ignorant to 'the facts of the data', making them uninferrable with respect to one of the most important distributional concepts, central tendency/location?

  • $\begingroup$ Entropy does not generally scale alongside variance, because the mapping from the random phenomenon on which entropy is defined to a random variable on which variance is defined can vary a lot. I can map a coin throw to a random variable $X$ with possible values $\{0,1\}$ or $Y$ with possible values $\{0,2\}$. The variance has quadrupled when moving from $X$ to $Y$, but the entropy of the coin throw stays the same. $\endgroup$ Aug 28 '20 at 14:17
  • 1
    $\begingroup$ @Richard The claim is not that entropy scales in direct proportion to the variance; it's a little subtler, as I explain in the linked thread. When $X$ is scaled (say by a factor $\sigma\gt 0$), both the variance and the exponential of the entropy change in simple, predictable ways that are strictly increasing functions of the scale factor: the variance is multiplied by $\sigma^2$ and the exponential entropy $\exp(H)$ is multiplied by $\sigma.$ This means, of course, that $H$ is shifted by $\log\sigma,$ strongly indicating there is no analog of the mean for $H.$ $\endgroup$
    – whuber
    Aug 28 '20 at 14:39
  • $\begingroup$ is information theory lacking something crucial by not having a match for the mean $\endgroup$
    – develarist
    Aug 28 '20 at 14:50
  • $\begingroup$ @whuber, I think my example illustrates that we cannot say anything about the relationship between entropy and variance unless we fix the mapping between the random phenomenon and the corresponding random variables. Your answer seems to apply conditionally on having fixed such a relationship. $\endgroup$ Aug 28 '20 at 14:58
  • 1
    $\begingroup$ @Richard There is no claim made that entropy and variance are universally related. For any given values $V$ of variance and $H$ of entropy corresponding to some distribution, scaling the distribution by $\lambda\gt 0$ changes the ordered pair $(\sqrt{V},\exp(H))$ into $\lambda (\sqrt{V},\exp(H)).$ Equivalently, the ratio $\exp(H)/\sqrt{V}$ is invariant under scaling. That's incontroverible, clear, insightful, and useful. The difference here is that you are discussing discrete distributions--but that's not relevant to the question, which explicitly assumes the distribution is continuous. $\endgroup$
    – whuber
    Aug 28 '20 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.