I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?

  • $\begingroup$ Entropy is a measure of information, not a statistic used to test a hypothesis. $\endgroup$ – ebb-earl-co Aug 4 '15 at 15:21
  • 1
    $\begingroup$ @ScouserInTrousers Could you elaborate on why this distinction important? If we can $t$-test that the difference in two heights is statistically significant, why can't we do the same for entropy? After all, "inches are just a measure of linear distance." $\endgroup$ – Sycorax Aug 4 '15 at 15:28
  • $\begingroup$ @user777 a $t$-test is comparing a value from the Student's $t$ distribution that was calculated from your data (assumed to be normal) to some critical value on the Student's $t$ distribution, which gives evidence to the plausibility of the hypothesis. In comparison, (relative) entropy is an estimate of the K-L distance between the hypothesized model and the true generating model (infinitely-complex underlying process in nature). There is no hypothesis tested, just an amount of information (yes, in the vague sense of bits) that is present in your explanation of the process (i.e. your model) $\endgroup$ – ebb-earl-co Aug 4 '15 at 15:33
  • 1
    $\begingroup$ @ScouserInTrousers It sounds like this is an answer to OP's question. Perhaps you should consider expanding on it and posting it so that you can garner additional reputation. $\endgroup$ – Sycorax Aug 4 '15 at 15:35
  • $\begingroup$ @ScouserInTrousers The amount of information in a sample is a random number. I guess I can infer if the KL-entropy value can be generated from pure chance or means something more. $\endgroup$ – emanuele Aug 4 '15 at 15:42

A fantastic reference that I have been using for self study on this topic is Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach by Kenneth Burnham. In brief, a hypothesis test compares a test statistic, $T$, calculated from the data on hand (assumed to be from some distribution, usually Gaussian) to a critical value, e.g. $t_{crit}$ or $F_{crit}$ or $\chi^2_{crit}$ to establish the veracity of a hypothesis, e.g. $\mu=0$.

Entropy, on the other hand, (better said, relative entropy) is an estimate of the expected Kullback-Leibler distance between the hypothesized model (could be Gaussian, Chi-square, whatever) and the true, generating model— i.e. nature— denoted by Burnham simply as $f$. What does estimated distance mean? Well, if the relative entropy, a.k.a. expected K-L distance between your candidate model, $g$, and full reality (nature) is large, then you have lost information (measured in bits) by using your candidate to try to represent reality. There is no hypothesis being tested in information-theoretic approaches, only (the distance from) truth to be discovered.

The quantity used in statistics most often to measure this distance is the AIC. A quote from the textbook on the AIC:

Thus, rather than having a simple measure of the directed distance between two models (i.e. the K-L distance), one has instead an estimate of the expected, relative distance between the fitted model and the unknown true mechanism (perhaps of infinite dimension) that actually generated the observed data. (Page 61)

  • $\begingroup$ As I said in my previous comment the KL-distance in a sample is random number. So I can test if this random number can be the outcome of pure chance or not. As chi square test does. $\endgroup$ – emanuele Aug 4 '15 at 15:49
  • $\begingroup$ @emanuele what you're skipping is the assumption that the data come from a $\chi^2$ distribution. If they do not, then a $\chi^2$ test is inappropriate. Also, the K-L distance is a random variable. If its distribution is not the $\chi^2$ distribution, then it doesn't make sense to compare an observed value of it to a $\chi^2$ critical value. $\endgroup$ – ebb-earl-co Aug 4 '15 at 15:55
  • $\begingroup$ I agree, but what I meaning is that you can know the distribution of the KL random values by simulation. Am I wrong? $\endgroup$ – emanuele Aug 4 '15 at 16:43
  • 1
    $\begingroup$ Thanx for the answer. I need some time to think about it. $\endgroup$ – emanuele Aug 4 '15 at 17:39
  • 1
    $\begingroup$ I don't see how this explains why one couldn't or shouldn't be interested in testing hypotheses about entropy. The discussion on KL divergence and AIC seems to be a distraction from this (particularly since KL divergence depends on cross entropy rather than ordinary entropy). $\endgroup$ – dsaxton Aug 6 '15 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.