Statistical methods are based on model assumptions. For example, an independent one-way ANOVA makes the following assumptions:
Normally distributed residuals
Homogeneity of variance
Independence of observations
Whether or not these assumptions are met will influence the reliability of the independent one-way ANOVA’s results and the conclusions drawn from them (to varying degrees, depending on what assumptions are violated and how).
My question is: When should we check the assumptions of our model? Is it preferable to first check model assumptions or inspect model fit? How might that influence the interpretations and decisions we make thereafter, and why might this be preferable to the other approach?
In the case of general linear models, we first need to fit our model, otherwise we cannot test whether the residuals are normally distributed. But immediately after that we could either choose to check model assumptions or inspect model fit.
In particular I am interested in answers that speak to any of the following three approaches:
The purpose of checking model assumptions is to decide whether the originally chosen test is appropriate for the data, so assumptions should be checked first. A different, more appropriate, test should be used if assumptions are violated, and conclusions should be drawn from this test. This approach is endorsed in textbooks (e.g., Dowdy et al., 2004), and is also the one I’ve encountered in statistics courses I’ve taken.
The purpose of checking model assumptions is to assess the quality of the model we originally chose in light of our data, so assumptions should be checked second. Depending on the severity of violations, conclusions drawn from test results might be reigned in, or the model might be respecified. This seems to be the approach endorsed by Fisher (see Spanos, 2017).
Model assumptions are often violated in the real world, so there’s no need to check them. Instead we should choose better default tests that are less constrained and stick with those (e.g., Declare et al., 2017, 2019). This is also a popular approach, for example, see Section 4 in the Spanos, 2017 paper cited above.
This preprint has some good discussion and examples comparing the performance of these approaches in different circumstances and with different tests. It concludes:
“In some setups either running a less constrained test or running the model-based test without preliminary testing have been found superior to the combined procedure involving preliminary [assumption checking to guide test selection].” However, “a sober look at the results reveals that the combined procedures are almost always competitive with at least one of the unconditional tests, and often with them both. It is clear, though, that recommendations need to depend on the specific problem, the specific tests involved. Results often also depend on in what way exactly model assumptions of the model-based test are violated, which is hard to know without some kind of data dependent reasoning.”
So the best performing approach depends on circumstances and the test used. However, if you were to pick one of the previously mentioned approaches as a general principle to follow when better information isn’t available, which would make the most preferable default?
Please base your answers on experience or evidence.