6
$\begingroup$

It is said that the distribution with the largest entropy should be chosen as the least-informative default. That is, we should choose the distribution that maximizes entropy because it has the lowest information content, allowing us to be maximally surprised. Surprise, therefore, is synonymous with uncertainty.

Why do we want that though? Isn't the point of statistics is to estimate with minimal error or uncertainty? Don't we want to extract the most information we can from a dataset/random variable and its distribution?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Hi: It's the least informative beforehand in the sense of making the least assumptions about the values of the distribution and still being subject to some constraint. ( the second moment being equal to $\sigma^2$). It's explained pretty nicely in the link below. The idea is that, BEFORE-HAND, you don't want to make any assumptions about the information in your sample. The distribution with the greatest entropy gives you that characterization. en.wikipedia.org/wiki/Maximum_entropy_probability_distribution. $\endgroup$
    – mlofton
    Commented Jul 29, 2020 at 11:52
  • $\begingroup$ ok at what point does the max entropy become useful though in the sense of minimizing uncertainty in characterizing a dataset? $\endgroup$
    – develarist
    Commented Jul 29, 2020 at 12:06
  • $\begingroup$ Maybe my answer at stats.stackexchange.com/questions/66186/… can help? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 29, 2020 at 20:04
  • $\begingroup$ i saw that before, it doesn't $\endgroup$
    – develarist
    Commented Jul 29, 2020 at 20:49
  • 2
    $\begingroup$ Who said the distribution with the largest entropy should be chosen as the least-informative default and in which context? $\endgroup$
    – Richard Hardy
    Commented Aug 20, 2020 at 14:58

1 Answer 1

Reset to default
7
$\begingroup$

Because "maxent" distribution is more "in the center". A formal description of this is in this paper -- "Game Theory, Maximum Entropy, Minimum Discrepancy, and Robust Bayesian Decision Theory". The basic idea if when you know some constraint is true, you better pick the maximum entropy distribution subject to this constraint, because it guarantees that you won't be too far from the worst-case true distribution (which could be hiding in the corner)

Here's an example -- spaces of all distributions over 3 outcomes with entropy contours

enter image description here

And here's the plot of entropy for all distributions. Picking highest entropy distribution gives you the one closest to the center, which also minimizes the distance (in the KL-divergence sense) to the furthermost point (aka, the potential true distribution)

enter image description here

One could visualize this in the original space with p1, p2, p3 being the 3 axes of the space of multinomial distributions over 3 outcomes

enter image description here

Improve this answer
$\endgroup$
9
  • $\begingroup$ is there a correspondence or some sort of corollary between the "center of a distribution" and the distribution's sample mean? $\endgroup$
    – develarist
    Commented Oct 7, 2020 at 21:03
  • 1
    $\begingroup$ "The center" here refers to center of the space of distributions. "Sample mean" is a moment, and there's a relationship between moments of distribution and being "close to the center" (equivalently, being "high entropy"). For instance E[XY]=E[X]E[Y] is one such relationship, your distribution has much higher entropy when this constraint is satisfied $\endgroup$
    – Yaroslav Bulatov
    Commented Oct 7, 2020 at 21:06
  • $\begingroup$ looking at the suggested paper, i didn't find the two images displayed in your answer. where did you get them? i'd like to confirm what the meaning and units of the 3 vertices and 3 axes are, and how the number of random variables used to form them can be inferred $\endgroup$
    – develarist
    Commented Oct 7, 2020 at 21:10
  • $\begingroup$ I made them myself. Thing of it as collection of all possible weighted 3-sided dice.The 3 vertices are 3 extremal distributions, (1,0,0), (0,1,0), (0,0,1), ie the dice lands on one of the faces with probability 1. The middle point is the dice where all sides are equally likely. Set of dice = set of multinomial distributions $\endgroup$
    – Yaroslav Bulatov
    Commented Oct 7, 2020 at 21:17
  • $\begingroup$ so the $z$-axis (vertical) is entropy? ticks between the vertices would help since I don't see how the dice outcomes mix across the 3 'variables', nor how entropy is being calculated at each 'grid-point' $\endgroup$
    – develarist
    Commented Oct 7, 2020 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.