When dealing with statistical tests, sometimes we can run into cases where many assumptions would apply and consequently would need to be tested. In complex models, testing many assumptions at 5% may in itself produce at least one significant test which may violate assumptions that would in turn prevent further testing.

Is there any rationale why we correct when performing post-hoc tests but not when testing assumptions even though the same problem applies in both situations in theory? Is it ever done and are there any recommendations?

  • 3
    $\begingroup$ Ordinarily, one might do one or two tests (e.g, normality, homoscedasticity) to check assumptions. If these tests are used to 'branch' to t vs. Wilcoxon, or to pooled vs. Welch t, there may be a question about the true P-value of the 'hybrid' (2-step) test. That is one reason some statisticians try to avoid using tests of assumptions in this way. (For example, by using Welch t tests when there is any question of heteroscedasticity.) // Formally, one usually seeks to avoid false discovery doing (up to 10) post hoc tests on pairs of levels of, say, a one-way ANOVA with 5 levels of the factor. $\endgroup$
    – BruceET
    Jan 31, 2021 at 21:26
  • 2
    $\begingroup$ Good points by @BruceET. Also, tests of assumptions tend to be a poor choice for checking model assumptions: They have poor power in the scenario when assumptions matter the most (small sample size), and are more likely to detect small (probably true) deviations that do not really matter (e.g. slightly non-normal residuals are known to be pretty harmless for e.g. randomized controlled trials with large samples sizes) for large sample sizes. Even "better", testing assumptions & changing analysis strategy based on it is known to inflate the type I error rate of the analysis you do in the end. $\endgroup$
    – Björn
    Feb 1, 2021 at 13:30

1 Answer 1

  1. Aris Spanos argues here and elsewhere that tests of assumptions (misspecification tests) should not be taken as affecting the later model-based inference, because what they are testing is essentially different from the later inference. This can be made mathematically precise in cases in which misspecification tests are run in a way that makes them statistically independent of the later inference, as in linear regression when doing misspecification testing on the residuals. In these cases running tests without correction is fine. In many other cases, misspecification tests may be approximately independent of the later inference, in which case misspecification testing can be treated as "mostly harmless". (This however does not automatically imply that it's a good thing to do, see below.)

  2. Standard correction, e.g., Bonferroni, of test levels is generally not a good idea in connection with misspecification testing, because rejection of a misspecification test is meant to indicate problems with the model assumptions rather than being treated as another "scientific discovery". Correction for multiple testing generally protects the level but will affect the power, however in this case affecting the power means that problems with the model may remain undetected. On the other hand, as we don't aim for rejecting the model assumption, there is little danger that a rejection of it will be overinterpreted other than by applying more caution when running the final inference. This is arguably tolerable even if ultimately the probabilty of a random rejection of any null hypothesis in the two-stage procedure involving the assumptions test may be larger than the nominal level. This is not a big problem if only the final hypothesis is of real scientific interest, so one can consider the level of that final test alone.

  3. To what extent misspecification testing is helpful is controversial. Unfortunately it depends strongly on what exactly is done, i.e., what misspecification test(s) you use, what model-based inference you want to run, what you do alternatively in case you find problems with the assumptions, and unfortunately also on how exactly the assumptions are violated in case they are violated (which of course the analyst cannot know). There are a number of studies that investigated procedures involving model assumption testing, and results are quite diverse. A survey is here. The authors also point out that although much work comes out critical about misspecification testing, and it sometimes is not a good idea indeed, the existing work overall may be too pessimistic, particularly because unless we can use subject matter knowledge, we don't have very good alternatives: Pretty much all problems with formal misspecification testing exist for informal alternatives like visual checking as well, and the dangers of using methods without checking assumptions are well known and documented.

As a side remark, some work investigating specific procedures involving misspecification tests recommends to run the misspecification test at a higher level than the usual 5%, which is the opposite of standard corrections.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.