How to measure how "good" or accurate a probability distribution is? Entropy, variance or what? How can one measure the accuracy of the probability distribution of, say, a physical magnitude?
I know one good candidate is the entropy, which measures the amount of information one has about the system---cf. Appendix A of Ref. [E. Jaynes, Physical Review 106, 620 (1957)]---.
However, one can also think of the variance as a way to evaluate uncertainty: the lower it is, the better we know the system.$^*$
Imagine now that one has two probability distributions of the same variable, $P_1$ and $P_2$.
Let $P_1$ have the lowest entropy and $P_2$ the lowest variance.
Is it even possible to say that one provides better knowledge of the system than the other?
Would another figure of merit different from the entropy and the variance answer such a question?
$^*$ As a side comment, the Heisenberg uncertainty principle is actually formulated in terms of the variance, but not the entropy of the probability distribution $|\psi(x)|^2$, $\psi(x)$ being the wavefunction.
 A: By definition, the notion of entropy is the one that gives you the amount of knowledge in a probability distribution (from Shannon information theory).
I would say that the variance is only one part of the puzzle regarding the knowledge. Indeed, for a probability distribution defined over $\mathbb{R}$ with a given variance, you have infinite possibilities of distributions, but the one with maximum entropy is the normal distribution (with that variance).
Regarding the accuracy provided by a distribution, this is a different question. Obviously the variance is a good metric but so are the higher order moments (especially the even ones).
So it all depends on what kind of metric you are trying to get.

Here's an example: suppose your distribution is defined over $[a, b]$, with no other knowledge.
The maximum entropy distribution is $\mathcal{U}(a,b)$. The variance of this distribution is $\frac{(b-a)^2}{12}$.
Now take a beta distribution with $\alpha$ and $\beta$ close to 0. The variance tends to a higher value, $\frac{(b-a)^2}{4}$ if I'm not mistaken, but it bears a much lower entropy, because any observation is very likely to be $a$ or $b$.
