A statistical test $T$ is a mapping from the space $\Delta$ of possible data $D$ to $\{R,A \}$, (meaning: Reject, Accept).

If we have a null hypothesis $H_0:\theta \in \Theta_0$ for which $T$ is a test, where $\theta$ is the parameter of the model that we assume generated data $D$, then the probability that $H_0$ is true is

$$P(H_0|D)=P(H_0)\cdot \frac {P(D|H_0)}{P(D)}$$

We can expand this into:

$$P(H_0|D)=\int_{\Theta_0}p(\theta)\cdot \frac {P(D|\theta)}{P(D)}d\theta$$

Now, what a statistical test essentially does, is it rejects $H_0$ if $D\in R$ (the rejection region, so $R\subset \Delta$), and "accepts" $H_0$ (or "does not reject" if you wish) if $D\in R^c$.

Now, when we do a test (or formulate a test statistic $t$), and make a judgment about $H_0$ on the basis of this, we are reducing $D$ to $t$ or to $R$. That is, we are removing information, and reducing equation $(1)$ to :

$$P(H_0|D)\approx P(H_0|t)=\int_{\Theta_0}p(\theta)\cdot \frac {p(t|\theta)}{p(t)}d\theta$$

Where $p(t|\theta)$ is the sample distribution of the test statistic.

My question is: Is there a term for the "extent to which we lose information" when reducing $D$ to a test statistic $t$ or to a rejection or non-rejection $R$? (if $t$ is a sufficient statistic for $H_0$, then this loss of information would be zero). This would be a way to evaluate specific tests, alongside the "power function" $p(R|\theta)$.

  • $\begingroup$ accepts" H0" (or "does not reject" if you wish) are not equivalent. Accepting H0 requires tests of concordance /equivalence, which define the tolerance within which you accept H0. The majority of hypothesis testing in scientific literature is testing for difference,the tolerance writing which you reject H0 $\endgroup$ – ReneBt Mar 30 '18 at 4:36
  • $\begingroup$ @ReneBt I agree from the viewpoint of absolute information content. That is if the data were an apple, the t-statistic would be an orange, that is, a t-statistic is not a condensation of the data, rather it is an index of contrast of $H0$ with data. $\endgroup$ – Carl Apr 2 '18 at 19:29
  • $\begingroup$ @Carl: The context of inference within a parametric model makes it clear that "information" here means information about the parameter $\theta$, & that's confirmed by the statement that the information loss is zero when the statistic $t$ is sufficient. Now, a scalar statistic is indupitably a reduction of the data vector; but that hardly precludes its also being "an index of contrast of $H_0$ with data", & the utility of such an index will depend on how much of the information about $\theta$ carried by the raw data is preserved by the reduction. $\endgroup$ – Scortchi - Reinstate Monica Apr 3 '18 at 14:06
  • $\begingroup$ @Scortchi My guess is that "an index of contrast of $H_0$ with the data" relates the context in which Fisher showed sufficiency, i.e., preservation of relative information content. A bit premature for me to say so, as I need to go over it in excruciating detail to be sure. If you want to chat about this, I would be up for it. $\endgroup$ – Carl Apr 4 '18 at 2:13
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Scortchi - Reinstate Monica Apr 4 '18 at 8:56

The term is simply "loss of information" (or "information loss"), & was introduced by Fisher shortly after he came up with the concept of sufficiency. See Fisher (1925), "Theory of Statistical Estimation", Proc. Camb. Philos. Soc., 22, pp 700 – 725. The usual way to measure it is in terms of Fisher information, the variance of the score:


where $\ell(\cdot)$ is the log-likelihood function. Given some regularity conditions


with equality only when $T$ is sufficient. So $\mathcal{I}_D(\theta)-\mathcal{I}_T(\theta)$ quantifies the extent to which information is thrown away by using the statistic $T$ for inference rather than the full data $D$.

Information loss is an important concept in conditional inference. If $(T,S)$ is jointly sufficient, while $S$ is ancillary, then the information lost by using $T$ for inference about $\theta$ can be recovered by conditioning on $S$:


See Fisher (1934), "Two New properties of Mathematical Likelihood", Proc. Royal Soc. Lond., A, 144, pp 285 – 307. Conditioning on an only approximate ancillary does result in information loss; an example is Fisher's Exact Test for a joint binomial sampling model, which is justified by the marginal totals' carrying only very little information about the odds ratio parameter. So information loss is a way to evaluate test statistics, though nugatory information loss doesn't imply high power—the power of FET is much reduced by the constriction of the sampling space owing to conditioning.

I'm not sure there's much point to considering the information loss in reducing the data to a accept/reject report according to a hypothesis test. A test $\phi(T)=\boldsymbol{1}_R(T)$ is a statistic too; but the log-likelihood function $\ell(\theta;\phi(T))$ is just the power function of the test (or its complement). Perhaps comparing information loss would be another way to decide between tests whose power functions cross.

† For any $\theta_1,\theta_2$ the Kullback – Leibler divergence from the distribution with density $f(x;\theta_1)$ to that with $f(x;\theta_2)$ is equal to the KL divergence from the distribution with density $f(t;\theta_1)$ to that with $f(t;\theta_2)$ only when $t$ is sufficient for $\theta$. Each divergence, given more regularity conditions, approaches the Fisher information as $\theta_1$ gets closer to $\theta_2$. See Kullback & Leibler (1951), "On information and sufficiency", Ann. Math. Statist., 22, 1, pp 79 – 86.

| cite | improve this answer | |

Simple answer to Q. Is there a term for the "extent to which we lose information" when reducing D to a test statistic t or to a rejection or non-rejection R? is that first that there is no simple loss of information and measuring it can be difficult and deceptive. Secondly, when we complete hypothesis testing we have optimistically defined that there is any information relating to our hypothesis, which has nothing to do with information loss, rather we are identifying latent information.

To define what information loss is, we need to first define what information is. For example, in some contexts (not the one being discussed here) there may be no information loss; noise reduction or image compression can be lossless. From this is should be obvious that information loss is context dependent. The elephant in the room is that there is no information without context. Consider Shannon's opinion on this, "Information processing may more specifically be defined in terms used by, Claude E. Shannon as the conversion of latent information into manifest information (McGonigle & Mastrian, 2011)."

To examine this question from a perspective of absolute information content requires quantification of information complexity, a difficult task, see Kolmogorov complexity. That is, absolute information content is the smallest encoding that can generate a data set. On the other hand, relative information content from Shannon entropy relates the amount of information to the natural logarithms (e.g. of 1's) of each of two datum, and as such is proportional to bit length of a random datum. Such things are applied to AIC via measuring relative entropy, K-L divergence (Edit: This applies to the other answer here, for which the connection between Fisher metric and the relative entropy indeed is a relation with K-L divergence). Although interesting, relative information content, is self-information in the self-comparative entropy context, and is not absolute entropy or absolute information content. Thus, as below, it does not really apply to the question asked.

For hypothesis testing Shannon entropy is not relevant, because we are testing for latent information; we are testing whether there is not information latent of a certain type ($H0$) within the total signal, where the total signal is noisy. For example, if we test for normality, and cannot reject normality as an outcome, then we cannot disprove that information is present, we have provided some clarity as to the presence of manifest information. Do we attribute absolute information content to noise? According to Wikipedia, Shannon would not, "Abstractly, information can be thought of as the resolution of uncertainty." For example, when we identify a mean and variance of a normal distribution, that distribution is completely specified to within a certain confidence interval, to be sure, but the point is that the residuals are then either unstructured and lack information in the Kolmogorov sense, or our evidence for normality is lacking.

Before going further, let us illustrate some of these concepts to show that they are necessary. Can absolutely random bits contain information? Consider cryptography. We can encrypt information in a random sequence of binary digits to yield a second string of random bits. The information that is potentially thus contained is upper limited by the bit length of the string. However, without the key, there is no information appreciated. Thus, information content is context dependent, no context (i.e., no key) means no manifest information. When we test hypotheses we are looking for a key. With apologies to those who cannot see out of two eyes and have trouble staring past two close images to fuse them into stereo vision, which is the "key" here, for a lots of fun physical illustration of pattern in randomness look at a few random dot stereograms on the net. In fact, here is an example that I created myself.

enter image description here

Again, to see this, you have to look at it cross-eyed, but level, so that you see three pictures. Concentrate on the middle picture, and you will see three vertical bands that seem to be closer to you than the screen.

The purpose of showing this is that it was generated from a few lines of code, so that the Kolmogorov complexity is small, even as an upper limit. Moreover, the code specifies a bit length to repetition that is exactly the product of all of the primes less than the number of bits generated, in this case 1230 primes for 10000 bits, and an encoding repetition of $5.953232305570463\times10^{4301}$ bit length, a much larger number than the $10^3$ bits encoded. That means that to crack this code using brute force, without a relative key, is not possible.

In this example, the Kolmogorov complexity is small, most of which generates our latent information (the hidden vertical bars). In the relative information context, the latent information is much smaller than the self-referential information of 10000 bits and the noise is perfect to within an astronomical context. For example, without the code itself, there is no cryptanalysis that will tell us whether the left or the right image is the encoding key, indeed no guarantee that both images are not encoded, so that the key is relative, and no current computer can crack much beyond a 64 bit encryption, never mind trying to crack a code whose bit encryption is always astronomically longer than the encoded number of bits. The number of 1's in both images is identical, there is no relative information content in a K-L divergence sense, i.e., the self-entropy is maximal and unchanged comparing images, unless comparative bit order is considered.

Suppose we calculate the mean and the median of a sampling of data. A first reaction to the difference between mean and median would be the argument that the median value discards everything that is not the median value itself (i.e., for an odd number sample size). Does it, now? To contrast a mean with a median, we are contrasting a parametric method with a ranking result, i.e., a nonparametric method.

Generally for nonparametric methods the power is less than it would be for a perfectly defined parametric solution of the same type, but typically if the parametric modelling is not excellently chosen, that would not be true. That is, a nonparametric method that ranks results rather than use ordinary distance calculations discards information, but in effect, the information loss is typically mild.

Real data is not very revealing, it has more noise than structure. Here is a data histogram displayed without information loss.

enter image description here

Now here is the same data put into wider histogram categories.

enter image description here

The naive view would be that we have lost a lot of information by putting the data into histogram categories; why in one category we now have the sum of 21 numbers, which destroys what those numbers were before addition. The more explanatory view is that in the new histogram, much of what we lost was noise, which noise in the Kolmogorov view is not information complexity, and which we needed to filter out in order to better appreciate what the signal, i.e., the information, was. That, is our "decoding key" is addition, and we did not lose much information by doing that, and in the process we increased our signal to noise ratio substantially. So too it is with hypothesis testing, what is important is not the potential for containing information measured as the bit length of the data, i.e., signal plus noise, rather we only count the bit lengths of latent information, i.e., actual signal as information.

With respect to the question, whatever the relative loss of information is, it is not relevant as we are testing as to whether there is a significant absence of information in the data that relates to the $H0$ context alone, and not to a larger context of all possible $H0$, and we do not lose information that we did not assume we had to begin with. If, on the other hand, we cannot show that latent information is absent, then we are interested in the type II error, the absence of false negative testing, and it is the power of the test (as per the other answer above), if it is good enough, that finally allows us to estimate that information was likely manifest. In terms of Kolmogorov complexity, there is information in $H0$, and when we test if the data has information of the same form as $H0$ using a t-statistic, we are not altering the information content, we are comparing that content with latent information in data. The t-statistic need not have the same complexity as either the data or $H0$ to accomplish that goal. Thus, the t-statistic can be thought of by analogy as the fulcrum of a balance beam, and the information content of a t-statistic itself is not representative of the data alone but rather is information concerning the contrast of the data with $H0$. In other words, if the data were an orange, the t-statistic would be an apple, two different things really.

| cite | improve this answer | |
  • $\begingroup$ I spent a good deal of time thinking this through. To those who downvote; leave a note rather than waste reputation points. $\endgroup$ – Carl Apr 1 '18 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.