1
$\begingroup$

If $X$ is a real random variable with a pdf, variance/standard deviation is a measure of $X$'s dispersion about the pdf's central tendency, which in turn is referred to as the mean of $X$. For many, variance acts as a measure of risk and uncertainty.

Shannon entropy is a measure of the unpredictability of $X$, and is frequently compared with and does have mathematical links to variance. The estimated entropy of $X$'s pdf has been shown to scale alongside variance, in that, as $X$ is transformed by scaling it with increasing variance, entropy will correspondingly shift or grow in value, indicating more spread in the distribution like how the increasing variance indicates more dispersion. Transformations of the mean, however, will not impact the entropy rate.

What is the information analogue of the mean then?

If there is none, does this show a fundamental weakness in information theory? Is entropy and mutual information's inability to distinguish between negative and positive source data for its input probabilities make these measures blind/ignorant to 'the facts of the data', making them uninferrable with respect to one of the most important distributional concepts, central tendency/location?

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ Entropy does not generally scale alongside variance, because the mapping from the random phenomenon on which entropy is defined to a random variable on which variance is defined can vary a lot. I can map a coin throw to a random variable $X$ with possible values $\{0,1\}$ or $Y$ with possible values $\{0,2\}$. The variance has quadrupled when moving from $X$ to $Y$, but the entropy of the coin throw stays the same. $\endgroup$
    – Richard Hardy
    Aug 28, 2020 at 14:17
  • 1
    $\begingroup$ @Richard The claim is not that entropy scales in direct proportion to the variance; it's a little subtler, as I explain in the linked thread. When $X$ is scaled (say by a factor $\sigma\gt 0$), both the variance and the exponential of the entropy change in simple, predictable ways that are strictly increasing functions of the scale factor: the variance is multiplied by $\sigma^2$ and the exponential entropy $\exp(H)$ is multiplied by $\sigma.$ This means, of course, that $H$ is shifted by $\log\sigma,$ strongly indicating there is no analog of the mean for $H.$ $\endgroup$
    – whuber ♦
    Aug 28, 2020 at 14:39
  • $\begingroup$ is information theory lacking something crucial by not having a match for the mean $\endgroup$
    – develarist
    Aug 28, 2020 at 14:50
  • $\begingroup$ @whuber, I think my example illustrates that we cannot say anything about the relationship between entropy and variance unless we fix the mapping between the random phenomenon and the corresponding random variables. Your answer seems to apply conditionally on having fixed such a relationship. $\endgroup$
    – Richard Hardy
    Aug 28, 2020 at 14:58
  • 1
    $\begingroup$ @Richard There is no claim made that entropy and variance are universally related. For any given values $V$ of variance and $H$ of entropy corresponding to some distribution, scaling the distribution by $\lambda\gt 0$ changes the ordered pair $(\sqrt{V},\exp(H))$ into $\lambda (\sqrt{V},\exp(H)).$ Equivalently, the ratio $\exp(H)/\sqrt{V}$ is invariant under scaling. That's incontroverible, clear, insightful, and useful. The difference here is that you are discussing discrete distributions--but that's not relevant to the question, which explicitly assumes the distribution is continuous. $\endgroup$
    – whuber ♦
    Aug 28, 2020 at 15:59

2 Answers 2

Reset to default
0
$\begingroup$

There are other measures of central tendency besides the mean. The median and mode are also measures of central tendency. Unlike the mean and median, the mode can be generalized to information theory: the value in the domain with the highest probability or probability density. So information theory does have an analog to central tendency, even if it does not have a direct analog to the mean.

The mean and variance are special-purpose tools designed for use with distributions over the real number line. As such, they incorporate assumptions intrinsic to the real number line such as the metric distance between two points. Information theory, on the other hand, constitutes a collection of general-purpose tools designed for use with distributions over arbitrary spaces. It can be applied to distributions over the real number line, but it just as well be applied to categorical data for which there is no intrinsic ordering. Inevitably, special-purpose tools will better fit the contexts for which they are designed than general-purpose tools.

So you're right that the tools provided by information theory don't necessarily stack up to the mean and variance when applied to real-valued data. But that doesn't mean that information theory is broken or incomplete.

In summary, use the tools that are best suited to the problem you are solving, and your concerns about them having relative weaknesses to each other will become moot.

Improve this answer
$\endgroup$
0
$\begingroup$

Variance is not similar to entropy at all. The variance has a scale, and entropy doesn't. Yes, the variance measures the uncertainty, but in a very different manner compared to the entropy. The variance, and especially its cousin standard deviation, reflect the absolute uncertainty. For instance, you could say that the volatility of S&P 500 index annual return is 20% in an average year. However, you can model the returns with different distributions, such as Gaussian or Student-t, then you'd get different entropies for the same reading of the volatility.

The mean also has a scale, the same scale as the dispersion (volatility). So you are looking for a metric that somehow reflects the location of the distribution but without a scale, i.e. relative to all other values. The only thing that comes to mind is skewness. It's a shape parameter, that places the location relative to the entire domain of the variable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.