Consider diagnostic testing of a fitted model, e.g. testing whether regression residuals are autocorrelated (a violation of an assumption) or not (no violation). I have a feeling that the null hypothesis and the alternative hypothesis in diagnostic tests often tend to be exchanged/flipped w.r.t. what we would ideally like to have.

If are interested in persuading a sceptic that there is a (nonzero) effect, we usually take the null hypothesis to be that there is no effect, and then we try to reject it. Rejecting $H_0$ at a sufficiently low significance level produces convincing evidence that $H_0$ is incorrect, and we therefore are comfortable in concluding that there is a nonzero effect. (There are of course a bunch of other assumptions which must hold, as otherwise the rejection of $H_0$ may result from a violation of one of those assumptions rather than $H_0$ actually being incorrect. And we never have 100% confidence but only, say, 95% confidence.)

Meanwhile, in diagnostic testing of a model, we typically have $H_0$ that the model is correct and $H_1$ than there is something wrong with the model. E.g. $H_0$ is that regression residuals are not autocorrelated while $H_1$ is that they are autocorrelated. However, if we want to persuade a sceptic that our model is valid, we would have $H_0$ consistent with a violation and $H_1$ consistent with validity. Thus the usual setup in diagnostic testing seems to exchange $H_0$ with $H_1$, and so we do not get to control the probability of the relevant error.

Is this a valid concern (philosophically and/or practically)? Has it been addressed and perhaps resolved?

  • $\begingroup$ Why do you think your autocorrelation test exchanges $H_0$ and $H_1$? If it fails to reject $H_0$, I would have thought your conclusion was that any autocorrelation you see in the residuals might have happened by chance so you go on using the model $\endgroup$ – Henry Jun 9 at 8:40
  • $\begingroup$ @Henry, I have no problem with the conclusion you describe. However, I think we want something else. We want to limit the probability of not rejecting a model when the assumption is violated to $\alpha$ (say, $\alpha=0.05$). But we do not actually do that. Instead, we limit the probability of rejecting a model under no violation of assumptions. I think the former is more relevant than the latter, hence the problem. In other words, I think nonrejection of a model when an assumption is violated should be type I error, but in the usual setup it happens to be type II error instead. $\endgroup$ – Richard Hardy Jun 9 at 9:43
  • $\begingroup$ That is moving toward power analysis. Otherwise, if you want there to be a small probability of using a model with no autocorrelation when there is in fact some underlying autocorrelation (even if it is tiny) then this could push you towards never using a model with no autocorrelation. Your choice; there are other cases where that kind of approach makes sense, such as never using an unpaired $t$-test assuming equal variances and instead always using Welch's test. $\endgroup$ – Henry Jun 9 at 9:53
  • $\begingroup$ @Henry, I think my question is more on the philosophical side. I start by motivating diagnostic testing from first principles and only then compare the ideal setup I have arrived at to the reality of diagnostic testing as currently observed in practice. So while power analysis may be a related topic, it is probably not the level on which the core of the discussion would be based. Though I may be mistaken. (And since you mentioned power analysis, the notion of severe testing by Mayo and Spanos might also be relevant.) $\endgroup$ – Richard Hardy Jun 9 at 9:57
  • $\begingroup$ This sounds like it is close to equivalence testing. In equivalence testing (I'm thinking of TOST), we show that the true difference (or autocorrelation for you) is within some amount of zero that we consider practically insignificant. We do not, however, show that the value is zero. $\endgroup$ – Dave Jun 9 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.