Why can we draw a more precise conclusion when we choose a lower accepted-risk in this hypothesis-testing setting, which seems contradictory?

Question

We want to know if 100 integer values (in a vector X) are following a Poisson $P(\lambda=2)$ distribution, which is our $H_0$ hypothesis.

Let's say the observed value of the $\chi^2$ with 5 degrees of freedom is $C=7.5$.

• With an accepted risk of $\alpha = 20\%$, the threshold for $\chi^2_5$ is $7.29 < C$. Thus we reject the hypothesis $H_0$. Conclusion: we don't know anything about the distribution of the vector $X$, except that it's probably not a Poisson of param. 2.

• With an accepted risk of $\alpha = 1\%$, the threshold for $\chi^2_5$ is $15.09 > C$. Thus we don't reject the hypothesis $H_0$. Conclusion: we now know that a good candidate for the distribution of $X$ is a Poisson distribution of parameter $2$.

How is it possible, in this $\chi^2$-goodness-of-fit setting, to:

• not know anything when we accept a high risk
• have a more precise idea when we decide to take a much lower risk!

This seems contradictory with the intuition: if we want a very low risk (1%), we should not be able to draw any conclusion at all here... And if we accept a higher risk (20%), we should be able to draw a conclusion.

Why is it the contrary here?

Note: I have read many questions/answers about Fisher / Neyman-Pearson, such as When to use Fisher and Neyman-Pearson framework? and others etc. but I think it would be useful to understand this precise example before being able to understand the whole theoretical setting.

Answer 1

What you are calling the "acceptable risk" here is more commonly known as the significance level of the test. In the case where you have a simple null hypothesis (as in your present test), this measures the probability of rejecting the null hypothesis conditional on the null hypothesis being true. In a shorthand sense, it is the conditional probability of a "false positive" in the test.

So, when you set the significance level $\alpha$ in this case, you are specifying the conditional probability of a false positive that you are willing to accept in your test. If you make this value smaller then you are saying that you want to reduce the conditional probability of a false positive. In order to achieve this reduction, the test is then adjusted to require more evidence for the alternative in order to reject the null hypothesis --- this makes it less likely that you will reject the null hypothesis, and therefore less likely that you will falsely reject it.

As you can see, when you chose the significance level $\alpha = 20\text{%}$ you rejected the null hypothesis. In this case you are willing to (incorrectly) reject the null hypothesis one-in-five times when it is actually true. Conversely, when you chose the smaller significance level $\alpha = 1\text{%}$, you are saying that you are now only willing to (incorrectly) reject the null hypothesis one-in-one-hundred times when it is actually true. Unsurprisingly, if you impose this higher level of caution with regard to a false positive, you need more evidence for the alternative to reject the null. In this case that higher level of evidence was not present, so you did not reject the null in this latter case.

Answer 2

Your question seems to be based on a mistaken premise.

Specifically: "Conclusion: we now know that a good candidate for the distribution of X is a Poisson distribution of parameter 2." is not a reasonable conclusion.

It may be a fairly poor candidate but perhaps you have little power to reject the null.

Failure to reject a null does not imply it's true (only that you lack sufficient evidence to conclude that it's false).

Failure to reject under a more stringent test even less so.

Note that the specific "risk" you're talking about should be made explicit -- it's the risk of incorrectly rejecting the null; that is, it's the chance -- when the null is true -- of rejecting it.

The higher you push that risk (making your rejection region larger) the better your chance of also rejecting a false null (i.e. the higher your power).

You can increase the rejection rate (both when correct and when incorrect) by having a larger rejection region and decrease it by having a smaller one. They move together.

You are (incorrectly) making a strong conclusion when you fail to reject, when the ability to say anything comes when you reject (i.e. that it's not consistent with a Poisson(2)).

It's a bit like a court case (at least in most English-speaking jurisdictions) -- if there's insufficient evidence then you can't convict, but that doesn't mean that the defendant was in fact innocent (for all that the law subsequently treats them as if they were). Absence of evidence is not the same thing as evidence of absence.

Goodness of fit tests don't tell you what you have, only (sometimes) that the data are not consistent with the thing under the null.