# Significance test for entropy?

Is there any way to test the difference of entropy given frequency table?

For example, let's say we have dice 1 and dice 2,

and we experimented with them and they showed up like

  die 1
num.    1  2  3  4  5  6
freq    6  7  3  5  2  1

die 2
num.    1  2  3  4  5  6
freq    3  4  2  1  1  2


The question is whether the entropy for die 1 and die 2 are different.

I thought chi-squared test for contingency table would do the work, but then again entropy is different in the aspect that the probability for two categories are interchanagable.

I mean that $$\text{Pr}(X=H)=0.2, \text{Pr}(X=F)=0.8$$ and $$\text{Pr}(X=F)=0.2, \text{Pr}(X=H)=0.8$$ have the same entropy!

So I wonder if there's any formal way of testing if the entropy is different 1...

• why do you want to test this? Commented May 28, 2019 at 8:14
• Would some kind of permutation test of bootstrap confidence interval of the difference in entropy well for you? (You can check if a $(1-\alpha)\%$ confidence interval contains $0$. If it does not, then that is equivalent to rejection of the null hypothesis at the level of $\alpha$, such as $\alpha=0.05$ making for a $95\%$ confidence interval.)
– Dave
Commented Jun 27, 2020 at 11:36

I agree with Nir Friedman that a Bayesian approach would be a good fit here, so I went ahead and implemented it in Python. Since the uniform prior is conjugate to the multinomial distribution, we can implement it without any fancy MCMC/HMC stuff. First things first, I imported a few libraries and defined a function to calculate entropy:

import numpy as np
import seaborn as sns
from matplotlib import pyplot as plt

def entropy(x):
return np.sum( -x*np.log2(x) , axis=-1)


Then, I took Monte-Carlo samples from the posterior distribution of the entropy. This was done in two steps:

1. First, notice that the posterior distribution for the true multinomial proportions is a Dirichlet. We can sample from it in Python in a single line of code: np.random.dirichlet(counts_die+1, 1000000)
2. Calculate the entropy for each sample from that Dirichlet distribution

The code for this is as follows:

counts_die_1 = np.array([6,7,3,5,2,1])
counts_die_2 = np.array([3,4,2,1,1,2])
entropy_die_1 = entropy(np.random.dirichlet(counts_die_1+1, 1000000))
entropy_die_2 = entropy(np.random.dirichlet(counts_die_2+1, 1000000))


Then, we can plot the distribution for the difference between the entropies:

sns.kdeplot(entropy_die_1-entropy_die_2, fill=True)
plt.axvline(0, ls="--", c="k")

# changing plot aesthetics
plt.gca().set(yticklabels=[], ylabel="", xlabel="Difference in entropy, in bits")
plt.gca().tick_params(left=False)
sns.despine(left=True)


The result looks like this:

We don't see evidence for a difference in entropies, and we can be fairly certain (>99%) that any such difference is less than half a bit. We can get the probability of die 1 being less random than die 2 like this:

(entropy_die_1 < entropy_die_2).mean()


This gives us 0.512942: very close to 0.50 meaning that we have little to no evidence of a die being more random than another.

I think what's a little tricky is this: in a Student T test for comparing whether two populations have the same sample mean, you have a null hypothesis that the two populations have the same sample mean. Notice that this is as far as you have to specify the null hypothesis, you don't need to specify what both means are for the null hypothesis, just the difference. Because the actual value of the two means does not affect the distribution that results in the t-quantity, only their difference.

Here, the situation is quite different. The null hypothesis that both entropies are 0, and the null hypothesis that both entropies are some other value, are very different. If the entropy is truly 0, then you should always measure zero entropy in every test (i.e. there's zero variance).

I would take a Bayesian approach instead: assuming uniform priors (on the multnomial distribution parameters, not the entropy) and a measured entropy, you can come up with the distribution of the true entropy using the likelihood function. Note that this basically boils down to a series of questions about the multinomial distribution and frequency counts. That is, you can do the problem by getting a distribution over the multinomial parameters by adding the probabilities over all the permutations of frequency counts (since they all yield the same entropy). Once you have that distribution, you can convert that to a distribution over entropy.

If you do that once for each die, then you'll two distributions. You can then get the distribution of the difference. You can see how sharply peaked that distribution is away from 0, and that will tell you about whether the difference in entropy is significant.

• I don’t think the null hypothesis would be that both entropies are zero. Wouldn’t it be that the difference in entropies is zero, same as the two-sample t-test has a null that the difference in means is zero, not that both distributions have means of zero?
– Dave
Commented Jun 27, 2020 at 11:33
• @Dave You completely misunderstood what I wrote. In a student T, the individual means have no impact on the distribution at all. So the null hypothesis that the difference in means is 0, gives you the whole distribution (subject to assuming the same fixed variance). So you can do your test without assuming anything about either mean. With entropies, you can assume the difference is 0, but that will not tell you the distribution; the distribution is still a function of what the entropy is, even if it's the same for both. Commented Jun 27, 2020 at 17:54

For each distribution, compute the maximum possible entropy as $$\log_2 N,$$ then divide by the actual entropy. Test this ratio using a Z-test for proportions.