Using asymptotic distribution under the null when the null hypothesis is false This is a general question about hypothesis testing in statistics. A standard hypothesis test as I see it is based on 3 things: 


*

*Stating a null hypothesis

*Computing a test statistic for the difference between the observed and the null hypothesis

*Comparing the test statistic using the asymptotic distribution under the null hypothesis


What I am wondering is whether it can be problematic to compare the test statistic with the asymptotic distribution under the null in the cases when the null hypothesis is false. This is often the case (for instance, in feature selection or misspecification testing).
Has anyone tested whether it is a reasonable approximation to simply use the asymptotic distribution under the null even if the null hypothesis is false? Can this be problematic or am I missing something?
 A: Let me bring up an analogy (one I've used before in a different context).
In this scene I am in a room which has some printers in it, with several colleagues. The room has no window. "I have to take some documents across the street shortly -- will it rain do you think?" I ask generally, as another colleague, Jim, walks into the room coming back from lunch; Jim goes over to a distant corner and fiddles with a printer.
Alison replies "No, I am pretty sure it won't be raining, since Jim always carries an umbrella if it looks like it might rain."
"But wait," Angela objects, "you can't rely on that reasoning. You said 'if it looks like it might rain' -- but then concluded that it won't rain. If it doesn't look like rain, then the reasoning you relied on is based on something that (likely) isn't the case. Isn't that a problem?"
Angela's objection may seem weird because we're used to reasoning with relations where the condition may be counterfactual (and we know it's logically valid to do so; this sort of reasoning is akin to modus tollens).
When Alison relied on an accepted relation of the form "if A then B" (we all accept that Jim's umbrella-carrying behavior is the case), and then used the fact that B didn't occur to conclude the falsity of the premise A, the fact that A was (likely to be) counterfactual doesn't make the conclusion based on the relation between A and B shaky.
It's similar with hypothesis testing.
If the null applies we expect particular behaviour from our test statistic. (Akin to the relation if it looks like rain, Jim will have an umbrella). 
If we observe something that would be exceptional under the hypothesis (it would be extraordinary for Jim not to have an umbrella if it seemed as if it would rain -- yet Jim has no umbrella), we conclude that the observed data (no umbrella) are inconsistent with the hypothesis that if we were outside it would seem likely to rain.
We should see an umbrella if it were about to rain (it would be extraordinary not to), but we don't; so we figure it's not about to rain. In hypothesis testing, we should see a test statistic drawn from a particular distribution if the null were true ... so when we see a sample statistic that would be extraordinary if the null were true, the data are not consistent with that null.
In the example below we have a test statistic of about 28, which is in the extreme tail of the distribution under the null. 

Since the point of the hypothesis testing is to figure out whether the data are consistent with the null, and what we see would be an extraordinary thing to observe if the null were true (but is quite consistent with the behavior of the test statistic under some of the situations that could pertain under the alternative), we 
conclude that what we observe is inconsistent with the null.
When we see an extraordinary value of the test statistic (extraordinary if the null were true) then either the null is true but we just had an amazingly rare event occur, or the null is not true and we don't need to rely on a miracle to explain what happened.
Its not problematic that $T_\text{obs}$ may have come from a distribution other than that under the null; indeed we want to rely on that as part of the logic of the test.
Consider instead what the circumstance would be if the test statistic had the same distribution when the null were true as when it was false. (This is like the case when Jim nearly always carries an umbrella, but does so completely regardless of the weather)
Whenever the test statistic was "unusual" we no longer have the above two explanations to choose between -- we have lost the "we don't need to rely on a miracle to explain" option that was available under the alternative. We are only left with "something unusual happened, we observed a statistic the like of which we should see only very rarely" --- here the data can tell us nothing of value about the hypothesis. We're not more likely to observed the rejection region when the null is true, so values in the rejection region don't help us. 
So rather than it being a problem, it's actually quite necessary to the way hypothesis tests are supposed to work.
