$H_0$ vs $H_1$ in diagnostic testing Consider diagnostic testing of a fitted model, e.g. testing whether regression residuals are autocorrelated (a violation of an assumption) or not (no violation). I have a feeling that the null hypothesis and the alternative hypothesis in diagnostic tests often tend to be exchanged/flipped w.r.t. what we would ideally like to have.
If are interested in persuading a sceptic that there is a (nonzero) effect, we usually take the null hypothesis to be that there is no effect, and then we try to reject it. Rejecting $H_0$ at a sufficiently low significance level produces convincing evidence that $H_0$ is incorrect, and we therefore are comfortable in concluding that there is a nonzero effect. (There are of course a bunch of other assumptions which must hold, as otherwise the rejection of $H_0$ may result from a violation of one of those assumptions rather than $H_0$ actually being incorrect. And we never have 100% confidence but only, say, 95% confidence.)
Meanwhile, in diagnostic testing of a model, we typically have $H_0$ that the model is correct and $H_1$ than there is something wrong with the model. E.g. $H_0$ is that regression residuals are not autocorrelated while $H_1$ is that they are autocorrelated. However, if we want to persuade a sceptic that our model is valid, we would have $H_0$ consistent with a violation and $H_1$ consistent with validity. Thus the usual setup in diagnostic testing seems to exchange $H_0$ with $H_1$, and so we do not get to control the probability of the relevant error.
Is this a valid concern (philosophically and/or practically)? Has it been addressed and perhaps resolved?
 A: The somewhat unsettling truth is that misspecification testing is not suitable for "persuading a skeptic that the model is valid". Generally, as you obviously understand, not rejecting the $H_0$ does not imply that the $H_0$ is true, and this is the case also in misspecification testing. What the test does is something weaker, namely it just tells you that certain observable problems with the model assumptions have not occurred. Still the misspecification test will not rule out that the data has been generated in a way that violate the model assumptions and may violate them badly. For example, an evil dependence structure could be at work that enforces the data to show a certain seemingly innocent pattern that you see even though this may be contrived enough to not look suspicious to your favourite test for independence (I'm not claiming that this is realistic, I'm just claiming that a misspecification test cannot rule out that this is technically possible).
Misspecification testing can to a certain extent reassure you, but it cannot secure model assumptions to be true.
Note that some would argue that the term "valid" is weaker than the term "true", and A. Spanos (2018) argues that if you do misspecification testing in the right way (i.e., testing all assumptions in a reasonable order, meaning that the misspecification test of one assumption is not sabotaged by the failure of another assumption), ultimately indeed you can be sure that the model is "valid" for the data, even though this doesn't mean it's "true". The way he does this is by defining the term "valid" basically as passing all those tests, because then, according to him, we know that the data looks like a typical realisation from the model. I think that this is misleading though, because as I have argued above, this does not rule out that in fact model assumptions are violated in harmful ways.
A message from this is that misspecification testing is never a substitute for thinking about the subject matter and the data generating process in order to know whether there are problems with the assumptions that you couldn't see from the data alone.
The following are additions that were made taking into account comments and discussion:

*

*In a comment, you already made reference to "severe testing" (Mayo and Spanos). Note that in their work you'll never find severity calculations that refer to misspecification tests, and for good reasons. Models can be violated in far too many and too complex ways in order to rule out all violations (or even just all relevant ones), and be it with a certain error probability.


*There's TOST as in the response by Dave. This can work if we focus on one particular assumption (for example an autocorrelation parameter $\alpha$ to be zero) and take everything else in the model specification for granted. And even then we can only reject $|\alpha|>c$ for some $c>0$ (how small $c$ can be will depend on the sample size); we cannot reject $\alpha\neq 0$.


*The original question was "how to choose the $H_0$", which I haven't really addressed up to now; instead of answering it, I will argue that we can't do much better than what is usually done. Remark 2 above is about an $H_0$ that isn't exactly the complement of the model assumption, rather rejecting it would secure (with the usual error probability) that the true $\alpha$ is close to zero, i.e., the model assumption. This is really the best we can hope for, and also it is not an accident that even this can only be achieved taking a host of other assumptions for granted. The thing is that we can never rule out too rich a class of distributions, because such a class will contain distributions that are so close (in case $\alpha\neq 0$) to the model assumption that they cannot be distinguished by any finite amount of data, or even distributions that are in terms of interpretation very different (like the "evil dependence structure" mentioned above), but can emulate perfectly whatever we observe, and can therefore not be rejected from the data. Famous early results in this vein are in Bahadur and Savage (1956) and Donoho (1988). Particularly there is no way to make sure that the underlying process has a density, let alone being normal or anything specific. (There is less work about evil dependence structures as far as I'm aware, because detecting them is outright hopeless.)


*Furthermore, the problem with TOST is that I'd suspect that this has a higher probability to reject a true model than the standard misspecification testing approach, and this is bad, because not only it would be a (type II) error, but also it will worsen the problem that running model-based analysis conditionally on the "correct" outcome of a misspecification test can be biased, as the theory behind standard analyses doesn't take MS-testing into account, see the Shamsudheen and Hennig arxiv paper for this issue and some more literature.
References:
Bahadur, R. and L. Savage (1956). The nonexistence of certain statistical procedures in nonparametric problems. Annals of Mathematical Statistics 27, 1115–
1122.
Donoho, D. (1988). One-sided inference about functionals of a density. Annals of
Statistics 16, 1390–1420.
Spanos A (2018) Mis-specification testing in retrospect. Journal of Economic Surveys 32:541–577
There's also this (with which I agree more):
M. Iqbal Shamsudheen, Christian Hennig (2020) Should we test the model assumptions before running a model-based test? https://arxiv.org/abs/1908.02218
A: This is a great question.  If we were to set autocorrelation as the null hypothesis we would have to be very specific about the type and amount.  If we reject this hypothesis we have not brought evidence against all types or amounts of autocorrelation, just the one we tested.  For this reason we set no autocorrelation as the null hypothesis, with the general alternative being some form and amount of autocorrelation.  This is in agreement with Henry's comment.  While I see a similarity between a diagnostic test and a TOST, these are not the same.  In a TOST we are hopeful to reject the null hypothesis in favor of the alternative.  In a diagnostic test we are hopeful for a failure to reject the null hypothesis.
We typically think of a small p-value as evidence against the null, reducing the null to the absurd, showing it is implausible.  By this same logic a large p-value could be seen as evidence in favor of the null (weak evidence against the null), showing it is not absurd, it is plausible.  Of course no hypothesis is proven false with a small p-value, nor is it proven true with a large one.  All we can do is provide the weight of the evidence.
There is no right or wrong for what hypothesis is considered the null and what is the alternative.  If you are using a Neyman-Pearson framework it is a matter of what you want as the default decision.  For instance, when investigating a treatment effect we often think of "no effect" as the null hypothesis.  However, in clinical development one might use a clinically meaningful effect as the null hypothesis (default decision) and only if there is sufficient evidence against this hypothesis would it be decided that the durg is not efficacious.  Under a Fisherian framework one would test all possible hypotheses to see the evidence against no treatment effect as well as evidence against a clinically meaningful effect.
A: I think it is exactly what the two one-sided tests procedure (TOST) does. TOST concedes that there might be some small effect but shows, with some level of confidence, that the effect is below the threshold of causing us to care. Perhaps there is a bit of autocorrelation, but an autocorrelation of $0.01$ might be effectively zero. If you truly want to show the value to be zero, not close to zero, with some confidence (credibility...), I cannot see a way to do it without going Bayesian and using a prior with $P(0)>0$. If you want to be frequentist, then I think the best you can do is to bound the value in a range.
(I do not have enough experience with Bayesian methods to have much of an opinion of using a prior that puts $P(\text{what we want})>0$, but that sure sounds like rigging the test.)
$$\text{TOST}\\
H_0: \vert\theta\vert\ge d\\
H_a: \vert\theta\vert<d$$
In this way, we flip the null and alternative hypothesis to show that the value of interest, $\theta$, is less than our tolerance for difference from zero, $d$.
There are equivalences between TOST and power calculations, so I think this satisfies your requirement for controlling power that you mentioned in your comment to Lewian.
A: I don't think your premise is accurate regarding model testing.  All the diagnostic tests for models that I am familiar with stipulate the model assumption as the null hypothesis and test for a departure from this that would falsify the assumption.  Even if we are talking to someone who is a skeptic of the model assumptions, the usual approach would be to show them that when we subject the model to diagnostic tests there is no evidence of any breach of the model assumptions; via tests where those assumptions are taken as the null hypothesis.
The problem with setting the null hypothesis as a violation of the model is that this is not a simple hypothesis --- it is a complex composite hypothesis that must stipulate the type and degree of the violation in assumptions (which would then beg the question of sensitivity analysis for the stipulated degree).
So, I am not convinced that there is any incongruity in the first place to resolve.
A: Aa
Hello mr. Hardy
I did read page, but cant comment so i post
Having autocorrelation in residuals for me is good thing (from usage stand point)- it give me "assurance" that info about next error term can be "known" from previous
I mean, many tests were designed to help solving real issues - and it works widely.
I cant see any problem, just predicting that model with H0 or H1 will be bad / good - i personally would still test both.
But different data, different approaches. I feel you try to broke things that are good on some issues, but bad on another... like saying "every h0 model is good and every h1 is bad" its like saying "all fruits are red" while clearly orange is orange.
Guessing some field is more approprieate to solve issue, usualy save initional time.
(like many stats tests for me ... they are good in their ways)
Its always about helping in some way, I hope it will help you help.
