# How to reject a distribution given a sample?

Let's assume that we have the following vary basic problem:

1. We have a set of $$N$$ real numbers sampled from an unknown distribution.
2. We have a distribution that we consider as a hypothesis.
3. We would like to know how probable is the given set of numbers, given the distribution.

For example, if we sample numbers from $$\mathcal{N}(0,1)$$ distribution and than assume that $$\mathcal{N}(10,10)$$ distribution is standing behind the data, we should be able to conclude that it is very unlikely to get what we got and, therefore we would reject the $$\mathcal{N}(10,10)$$ distribution.

So, my question is: How can we calculate this "probability"? Or, in other words, what would be the measure of consistency between a distribution and a sample?

The first answer that comes to my mind is to calculate likelihood. However, I have several concerns with this approach:

1. Likelihood is not probability and, therefore, it is not clear how to interpret the calculated values. What value should we consider small enough to reject the hypothesis?
2. A "good" likelihood does not mean that we have an agreement between the distribution and the sample. For example if we sample 1000 values from $$\mathcal{N}(0,1)$$ distribution and calculate likelihood of this data set using the same $$\mathcal{N}(0,1)$$ distribution, we probably get a "good" value of likelihood (not too small). But if we make all 1000 observations positive (just by taking absolute value of observations), than it will be obvious that $$\mathcal{N}(0,1)$$ distribution is not an adequate description of the data set anymore (we never have observed negative numbers) but likelihood of this data set will be exactly the same as before if we assume $$\mathcal{N}(0,1)$$ distribution!
• Maybe just use the average. Then either a p-value or a Bayesian posterior probability based on the average might work. Dec 23, 2022 at 13:29

We would like to know how probable is the given set of numbers, given the distribution.

For a continuous distribution, this probability will be zero.

You appear to seek some test statistic that will be different when the distribution from which the sample was drawn is not the hypothesized distribution than when it is. If yuo approach it this way you can then create sets of nested rejection rules (and hence obtain p-values, which appears to be essentially what you're asking for, but now without the problem that getting exactly the observed statistic will have a probability of 0)

You can use the likelihood as the basis of a statistic for this, though it often ends up being not especially attractive as an omnibus test (there's often some set of alternatives of interest that it would have low power against). For example, consider an exponential model with mean $$\mu$$. The negative of the log of the likelihood is

$$-\log L = n \log(\mu) + n\frac{\bar{x}}{\mu}$$

and a linear rescaling of that is:

$$-\log L/n - \log(\mu) = \frac{\bar{x}}{\mu}$$

The right hand side is a pivotal quantity.

Now $$\bar{X} \sim\text{Gamma}(n,\mu/n)$$ (in the shape-scale parameterization). Under $$H_0$$, $$\frac{\bar{X}}{\mu_0} \sim\text{Gamma}(n,1/n)$$. If, with fixed $$\mu_0$$ we reject samples with small likelihood, we will reject samples with large sample mean relative to the hypothesized mean. This Gamma distribution under the null yields an exact test at any sample size.

Consider now that any distribution for which the CLT would apply and which has the same mean and standard deviation will converge to the same large sample distribution under the null (so we wouldn't expect good power against those distributions), but it will also tend to completely miss distributions with small means. Neither outcome seems ideal. It will, however, be truly excellent at picking up exponentially distributed data with a too-large mean.

This is more or less common with likelihood as a goodness of fit statistic -- it's great at discriminating against some pretty particular kinds of alternatives and not so great at others.

It seems important, then, to either consider which particular alternatives we seek power against (and choose a test statistic accordingly*), or to construct a test statistic with at least reasonable power against a wide variety of alternatives.

The latter path leads to omnibus goodness of fit tests, of which there are many alternatives.

* in particular, if you have a specific alternative in mind (similar to the situation you seem to discuss in your question), see the Neyman-Pearson lemma. If you have a sequence of alternatives governed by a parameter, see the Karlin-Rubin theorem.