Is "Information" somehow Related to "Variance"?

Question

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information" (i.e. constraints) is available about some class of probability distribution function (e.g. domain over which the probability function is defined, expectation, etc.), we can use the principle of Maximum Entropy to determine the "most informative" probability distribution function from this class of probability distribution functions in this situation.

Apparently, in many real world situations (e.g. when the data is continuous and can take any value between negative infinity and positive infinity) - the Normal Distribution ends up being the probability distribution function with the Maximum Entropy, thus often resulting in the "most informative" choice of probability distribution function when compared to any other candidate.

My Question (This is an obvious question): Why is the "most informativeness property" useful when selecting probability distributions?

In a casual sense, we can loosely assume that "more information" tends to be better and more useful than "less information". For example, a sign containing information about elephants at the zoo contains less information than an entire book dedicated to elephants. It is very likely that the book about elephants contains all the information present on the sign at the zoo, but the book about elephants also contains a lot of information about elephants that is not on this sign. Thus, we could reasonably conclude that "book with more information about elephants is more useful than the sign at the zoo with less information about elephants".

However, in the case of Maximum Entropy and the "Most Informativeness Property" of the normal distribution, why is this property useful? I am guessing that the "Most Informativeness Property" of the normal distribution somehow "naturally" resulted in "more successful applications" (e.g. real world statistical models with higher consistency, higher accuracy and lower variance) and in turn made it more "popular" and "favored".

I am guessing that perhaps there might be some relationship between Maximum Entropy, Informativeness and Variance?

Thanks!

Answer 1

Why is the "most informativeness property" useful when selecting probability distributions?

It isn't. It relates to the principle of maximum entropy, that

... is based on the premise that when estimating the probability distribution, you should select that distribution which leaves you the largest remaining uncertainty (i.e., the maximum entropy) consistent with your constraints. That way you have not introduced any additional assumptions or biases into your calculations.

Entropy measures the "surprise" value of obtaining additional information or "ignorance" (Sivia and Skilling, 2006). By making such a choice, you maximize your chances of learning something from the data (vs running into a confirmation bias trap). You may also find interesting the What is the role of the logarithm in Shannon's entropy? thread that explains how entropy works.

Now, commenting on other parts of the question:

the Normal Distribution ends up being the probability distribution function with the Maximum Entropy, thus often resulting in the "most informative" choice of probability distribution function when compared to any other candidate.

Again, the opposite: it is the "least informative". This relates to the concept of the maximum entropy distribution. Also, notice that normal distribution is not "the" distribution that maximizes entropy, but one of many maximum entropy distributions, each having this property in a specific scenario.