Does the problem of multiple testing also apply to the testing of assumptions?

Question

When dealing with statistical tests, sometimes we can run into cases where many assumptions would apply and consequently would need to be tested. In complex models, testing many assumptions at 5% may in itself produce at least one significant test which may violate assumptions that would in turn prevent further testing.

Is there any rationale why we correct when performing post-hoc tests but not when testing assumptions even though the same problem applies in both situations in theory? Is it ever done and are there any recommendations?

Answer 1

1. Aris Spanos argues here and elsewhere that tests of assumptions (misspecification tests) should not be taken as affecting the later model-based inference, because what they are testing is essentially different from the later inference. This can be made mathematically precise in cases in which misspecification tests are run in a way that makes them statistically independent of the later inference, as in linear regression when doing misspecification testing on the residuals. In these cases running tests without correction is fine. In many other cases, misspecification tests may be approximately independent of the later inference, in which case misspecification testing can be treated as "mostly harmless". (This however does not automatically imply that it's a good thing to do, see below.)

2. Standard correction, e.g., Bonferroni, of test levels is generally not a good idea in connection with misspecification testing, because rejection of a misspecification test is meant to indicate problems with the model assumptions rather than being treated as another "scientific discovery". Correction for multiple testing generally protects the level but will affect the power, however in this case affecting the power means that problems with the model may remain undetected. On the other hand, as we don't aim for rejecting the model assumption, there is little danger that a rejection of it will be overinterpreted other than by applying more caution when running the final inference. This is arguably tolerable even if ultimately the probabilty of a random rejection of any null hypothesis in the two-stage procedure involving the assumptions test may be larger than the nominal level. This is not a big problem if only the final hypothesis is of real scientific interest, so one can consider the level of that final test alone.

3. To what extent misspecification testing is helpful is controversial. Unfortunately it depends strongly on what exactly is done, i.e., what misspecification test(s) you use, what model-based inference you want to run, what you do alternatively in case you find problems with the assumptions, and unfortunately also on how exactly the assumptions are violated in case they are violated (which of course the analyst cannot know). There are a number of studies that investigated procedures involving model assumption testing, and results are quite diverse. A survey is here. The authors also point out that although much work comes out critical about misspecification testing, and it sometimes is not a good idea indeed, the existing work overall may be too pessimistic, particularly because unless we can use subject matter knowledge, we don't have very good alternatives: Pretty much all problems with formal misspecification testing exist for informal alternatives like visual checking as well, and the dangers of using methods without checking assumptions are well known and documented.

As a side remark, some work investigating specific procedures involving misspecification tests recommends to run the misspecification test at a higher level than the usual 5%, which is the opposite of standard corrections.