Significance test for entropy? Is there any way to test the difference of entropy given frequency table?
For example, let's say we have dice 1 and dice 2,
and we experimented with them and they showed up like
  die 1
  num.    1  2  3  4  5  6
  freq    6  7  3  5  2  1

  die 2
  num.    1  2  3  4  5  6
  freq    3  4  2  1  1  2

The question is whether the entropy for die 1 and die 2 are different.
I thought chi-squared test for contingency table would do the work,
  but then again entropy is different in the aspect that
  the probability for two categories are interchanagable.
I mean that $\text{Pr}(X=H)=0.2, \text{Pr}(X=F)=0.8$ and $\text{Pr}(X=F)=0.2, \text{Pr}(X=H)=0.8$ have
  the same entropy!
So I wonder if there's any formal way of testing if the entropy is different 1...
 A: I think what's a little tricky is this: in a Student T test for comparing whether two populations have the same sample mean, you have a null hypothesis that the two populations have the same sample mean. Notice that this is as far as you have to specify the null hypothesis, you don't need to specify what both means are for the null hypothesis, just the difference. Because the actual value of the two means does not affect the distribution that results in the t-quantity, only their difference.
Here, the situation is quite different. The null hypothesis that both entropies are 0, and the null hypothesis that both entropies are some other value, are very different. If the entropy is truly 0, then you should always measure zero entropy in every test (i.e. there's zero variance). 
I would take a Bayesian approach instead: assuming uniform priors (on the multnomial distribution parameters, not the entropy) and a measured entropy, you can come up with the distribution of the true entropy using the likelihood function. Note that this basically boils down to a series of questions about the multinomial distribution and frequency counts. That is, you can do the problem by getting a distribution over the multinomial parameters by adding the probabilities over all the permutations of frequency counts (since they all yield the same entropy). Once you have that distribution, you can convert that to a distribution over entropy.
If you do that once for each die, then you'll two distributions. You can then get the distribution of the difference. You can see how sharply peaked that distribution is away from 0, and that will tell you about whether the difference in entropy is significant. 
A: For each distribution, compute the maximum possible entropy as log2N, then divide by the actual entropy. Test this ratio using a Z-test for proportions.
A: I agree with @Nir Friedman that a Bayesian approach would be a good fit here, so I went ahead and implemented it in Python. Since the uniform prior is conjugate to the multinomial distribution, we can implement it without any fancy MCMC/HMC stuff. First things first, I imported a few libraries and defined a function to calculate entropy:
import numpy as np
import seaborn as sns
from matplotlib import pyplot as plt

def entropy(x):
  return np.sum( -x*np.log2(x) , axis=-1)

Then, I took Monte-Carlo samples from the posterior distribution of the entropy. This was done in two steps:

*

*First, notice that the posterior distribution for the true multinomial proportions is a Dirichlet. We can sample from it in Python in a single line of code: np.random.dirichlet(counts_die+1, 1000000)

*Calculate the entropy for each sample from that Dirichlet distribution

The code for this is as follows:
counts_die_1 = np.array([6,7,3,5,2,1])
counts_die_2 = np.array([3,4,2,1,1,2])
entropy_die_1 = entropy(np.random.dirichlet(counts_die_1+1, 1000000))
entropy_die_2 = entropy(np.random.dirichlet(counts_die_2+1, 1000000))

Then, we can plot the distribution for the difference between the entropies:
sns.kdeplot(entropy_die_1-entropy_die_2, fill=True)
plt.axvline(0, ls="--", c="k")

# changing plot aesthetics
plt.gca().set(yticklabels=[], ylabel="", xlabel="Difference in entropy, in bits")
plt.gca().tick_params(left=False)
sns.despine(left=True)

The result looks like this:


We don't see evidence for a difference in entropies, and we can be fairly certain (>99%) that any such difference is less than half a bit. We can get the probability of die 1 being less random than die 2 like this:
(entropy_die_1 < entropy_die_2).mean()

This gives us 0.512942: very close to 0.50 meaning that we have little to no evidence of a die being more random than another.
Hope it was helpful!
