Using asymptotic distribution under the null when the null hypothesis is false

This is a general question about hypothesis testing in statistics. A standard hypothesis test as I see it is based on 3 things:

1. Stating a null hypothesis
2. Computing a test statistic for the difference between the observed and the null hypothesis
3. Comparing the test statistic using the asymptotic distribution under the null hypothesis

What I am wondering is whether it can be problematic to compare the test statistic with the asymptotic distribution under the null in the cases when the null hypothesis is false. This is often the case (for instance, in feature selection or misspecification testing).

Has anyone tested whether it is a reasonable approximation to simply use the asymptotic distribution under the null even if the null hypothesis is false? Can this be problematic or am I missing something?

• The entire point of the hypothesis test is to try to figure out when the null hypothesis is false. You figure out the distribution given the null is true, and if it is instead false the test statistic will be from a different distribution where it will tend to have a higher chance of being in the rejection region. So any kind of hypothesis test will tend to be done in situations where the null may be false. Can you clarify what's different here? Jan 24, 2016 at 11:54
• There's nothing different here - I'm also discussing doing hypothesis testing in situations where the null may be false. What I'm wondering is whether it is problematic to define the rejection region based on a distribution that we know isn't the correct distribution.
– pir
Jan 24, 2016 at 15:51
• This question may be challenging to answer because it appears to be founded on one or more misconceptions. First, the null hypothesis testing framework has nothing to do with asymptotic distributions. Second, the very form of the question suggests it is based on an unconventional understanding of how a hypothesis test works. I suspect that reviewing the standard accounts of the subject might be the best way to address this question.
– whuber
Jan 24, 2016 at 17:36

Let me bring up an analogy (one I've used before in a different context).

In this scene I am in a room which has some printers in it, with several colleagues. The room has no window. "I have to take some documents across the street shortly -- will it rain do you think?" I ask generally, as another colleague, Jim, walks into the room coming back from lunch; Jim goes over to a distant corner and fiddles with a printer.

Alison replies "No, I am pretty sure it won't be raining, since Jim always carries an umbrella if it looks like it might rain."

"But wait," Angela objects, "you can't rely on that reasoning. You said 'if it looks like it might rain' -- but then concluded that it won't rain. If it doesn't look like rain, then the reasoning you relied on is based on something that (likely) isn't the case. Isn't that a problem?"

Angela's objection may seem weird because we're used to reasoning with relations where the condition may be counterfactual (and we know it's logically valid to do so; this sort of reasoning is akin to modus tollens).

When Alison relied on an accepted relation of the form "if A then B" (we all accept that Jim's umbrella-carrying behavior is the case), and then used the fact that B didn't occur to conclude the falsity of the premise A, the fact that A was (likely to be) counterfactual doesn't make the conclusion based on the relation between A and B shaky.

It's similar with hypothesis testing.

If the null applies we expect particular behaviour from our test statistic. (Akin to the relation if it looks like rain, Jim will have an umbrella).

If we observe something that would be exceptional under the hypothesis (it would be extraordinary for Jim not to have an umbrella if it seemed as if it would rain -- yet Jim has no umbrella), we conclude that the observed data (no umbrella) are inconsistent with the hypothesis that if we were outside it would seem likely to rain.

We should see an umbrella if it were about to rain (it would be extraordinary not to), but we don't; so we figure it's not about to rain. In hypothesis testing, we should see a test statistic drawn from a particular distribution if the null were true ... so when we see a sample statistic that would be extraordinary if the null were true, the data are not consistent with that null.

In the example below we have a test statistic of about 28, which is in the extreme tail of the distribution under the null.

Since the point of the hypothesis testing is to figure out whether the data are consistent with the null, and what we see would be an extraordinary thing to observe if the null were true (but is quite consistent with the behavior of the test statistic under some of the situations that could pertain under the alternative), we conclude that what we observe is inconsistent with the null.

When we see an extraordinary value of the test statistic (extraordinary if the null were true) then either the null is true but we just had an amazingly rare event occur, or the null is not true and we don't need to rely on a miracle to explain what happened.

Its not problematic that $T_\text{obs}$ may have come from a distribution other than that under the null; indeed we want to rely on that as part of the logic of the test.

Consider instead what the circumstance would be if the test statistic had the same distribution when the null were true as when it was false. (This is like the case when Jim nearly always carries an umbrella, but does so completely regardless of the weather)

Whenever the test statistic was "unusual" we no longer have the above two explanations to choose between -- we have lost the "we don't need to rely on a miracle to explain" option that was available under the alternative. We are only left with "something unusual happened, we observed a statistic the like of which we should see only very rarely" --- here the data can tell us nothing of value about the hypothesis. We're not more likely to observed the rejection region when the null is true, so values in the rejection region don't help us.

So rather than it being a problem, it's actually quite necessary to the way hypothesis tests are supposed to work.

• Excellent answer! I think many introductory statistic classes could benefit from such an explaination.
– pir
Jan 25, 2016 at 1:08