1
$\begingroup$

Related question here

I am trying to measure the uniformity of multimodal distributions and am looking into using entropy.

I would like a measure of entropy that is higher for the first distribution than the second.

However, Shannon entropy is order independent (invariant under permutations) so the above distributions have the same Shannon entropy.

Is there anyway to capture clumping in the measure of entropy?

My thinking is that under the heat equation, it should take longer for the second distribution's Shannon entropy to become asymptotic so it should have lower entropy.


Let there be 6 bins evenly filled out of 12. Shannon entropy doesn't distinguish between the case A where all even bins are filled vs case B where bins [0,3) and [9, 12) are filled

index 0 1 2 3 4 5 6 7 8 9 10 11
A 0 1/6 0 1/6 0 1/6 0 1/6 0 1/6 0 1/6
B 1/6 1/6 1/6 0 0 0 0 0 0 1/6 1/6 1/6

Right--but so what? The indexes of the bins are irrelevant. If they are relevant to you, then you do not need entropy; you need another concept. But in that case, what are you trying to characterize? @whuber

I though this might have been studied and you could tell me?

$\endgroup$
7
  • 2
    $\begingroup$ Could you write those pictures out as distributions? $\endgroup$ Sep 23, 2022 at 23:17
  • $\begingroup$ @conjectures I hope my edit is good enough. I couldn't get markdown tables to work so I've given up $\endgroup$ Sep 23, 2022 at 23:27
  • 1
    $\begingroup$ The two distributions you illustrate do not have the same entropy. $\endgroup$
    – whuber
    Sep 23, 2022 at 23:37
  • 1
    $\begingroup$ @Galen thanks, it was just the preview that was broken so I couldn't see what I was doing $\endgroup$ Sep 23, 2022 at 23:38
  • 2
    $\begingroup$ This has been asked before, see stats.stackexchange.com/questions/483535/… There are ideas to define entropies taking into account distances $\endgroup$ Sep 24, 2022 at 0:34

2 Answers 2

2
$\begingroup$

Shannon's entropy distinguishes between those two scenarios:

Two bins:

$$- \sum_{i=1}^2 \frac{1}{2} \log \frac{1}{2} = \log 2$$

Ten bins:

$$- \sum_{i=1}^{10} \frac{1}{10} \log \frac{1}{10} = \log 10$$

Finite $n$ bins:

$$- \sum_{i=1}^{n} \frac{1}{n} \log \frac{1}{n} = \log n$$

And note that $\log 10 > \log 2$. And since logarithms are monotonic functions, we know that more bins means a bigger number. I.e. $m > n \implies \log m > \log n$.


You are right that Shannon's entropy is invariant to the permutations of the bins. If the entropy depended on permutation, then it would distinguish between different versions of your two examples. You have ten bins in the first example. Such a permutation-varying entropy would consider the order of bins (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) different from (1, 2, 3, 4, 5, 6, 7, 8, 10, 9), and yet different from (1, 2, 3, 4, 5, 6, 7, 9, 8, 10),.... for all $10! = 3628800$ permutations of ten bins.


Re: 10/20 bins filled example:

You might be interested in normalizing the entropy to the max possible entropy of 20 bins.

Let us assume there is 20 outcomes, then for any uniform choice on ten of those bins we will have a normalized entropy of:

$$0 \leq \frac{H_{10}}{H_{\max}} = \frac{\log 10}{ \log 20} = \log_{20}(10) \approx 0.768621786840241 \leq 1$$


Is there anyway to capture clumping in the measure of entropy?

I'm not sure this is really what you want, but you could compute a sort of weighted entropy:

$$H_{w(x)} = - \sum_x [w(x)p(x)] \log [w(x)p(x)]$$

where $0 \leq w(x) \leq 1$ is some kind of weight based on the distances between bins which satisfies $\sum_x w(x) = 1$.

$\endgroup$
6
  • $\begingroup$ Let there be 10 bins evenly filled out of 20. Shannon entropy doesn't distinguish between the case where all even bins are filled vs where bins [0,5) and [15, 20) are filled $\endgroup$ Sep 23, 2022 at 23:17
  • 1
    $\begingroup$ Right--but so what? The indexes of the bins are irrelevant. If they are relevant to you, then you do not need entropy; you need another concept. But in that case, what are you trying to characterize? $\endgroup$
    – whuber
    Sep 23, 2022 at 23:38
  • $\begingroup$ Thanks for you help, I guess this the answer is that there is no knowledge on this because when statisticians looked into this they ended up in a dead end. I think I'll open a new question about another approach to the problem $\endgroup$ Sep 23, 2022 at 23:54
  • 2
    $\begingroup$ @TomHuntington: Have a look at stats.stackexchange.com/questions/483535/… $\endgroup$ Sep 24, 2022 at 0:52
  • $\begingroup$ @kjetilbhalvorsen thanks I am $\endgroup$ Sep 24, 2022 at 0:54
1
$\begingroup$

Variance can be generalized to be about any point rather than just the mean (i.e. how spread out is the distribution from this point) $$\mu \mapsto\sum p_i (x_i - \mu)^2.$$

We can also "invert" the variance to get a similarity measure for each point (how bunched is the distribution about this point) $$\mu \mapsto \sum p_i e^{-(x_i - \mu)^2}.$$

Can then take the log of this to get a surprise function $$\sigma :=\mu \mapsto -\log\sum p_i e^{-(x_i - \mu)^2}.$$

Entropy is then given by the formula $$\prod \sigma(x_i)^{-p_i}$$ as explained by blog post 1 and 2 given by @kjetil answer. These links do a very good job of developing the language you need to think about entropy.

  • Abundance: the proportions in which the species occur (e.g. 50% grass, 30% clover, 20% daisies)
  • Similarity: the extent to which the species are related (e.g. an ecosystem made up of 50% snails and 50% slugs should probably be regarded as less diverse than one made up of 50% snails and 50% bats).

Shannon entropy is only a good measure when working with discrete metric spaces i.e. the distance between different "bins" is infininte. However, our bins have a natural distance between them (i.e. the difference of the indices) and so we need something more

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.