How to handle assumptions of statistical tests?

Question

Does any definitive work exist, e.g. a review, a book chapter, or a book, on the advantages and disadvantages of existing approaches? Is there a consensus on which approach is the right one?

I have come across three approaches so far:

1. We test by a statistical test whether a given assumption of a statistical test is met. The problem with this is that the error of checking the assumption actually adds to the error of the test we originally wanted to perform. So it's misleading to report later the result of the latter test as, e.g. $p<0.05$ because, in fact, the test procedure's error rate is more than $5\%$. Is this accumulation of errors calculable?

2. Later I was taught that the previous approach is heresy. Instead, we tested by Q-Q plots the probably most common assumption: normality. But it is also based on the sample on which we will later apply the test we want to perform. Is this relevant? Plus, it seems like all that’s happening is that we replace the statistical error with a subjective error because the figure needs to be assessed. This subjective error is certainly not calculable. It might be decreased by randomly generating as many data points as there are in our sample from a normal distribution parametrized based on the sample mean and SD, and comparing the Q-Q plots.

3. We don't use parametric tests. Well, we use them only if we know a priori that their assumptions are met. When we don't know that for sure, we don't check anything, we use non-parametric tests.

Answer 1

In my opinion semiparametric regression models provide the best default approach. They are invariant to Y-transformations and contain common nonparametric tests as special cases, but allow for much more than that. An overview is here.

As you rightly noted, assessing assumptions (whether by p-values or graphs) can make unknown changes to tests' operating characteristics.

Answer 2

Your question is a laundry list of suggestions for doing one-sample tests of location, not an explicit question. So this is a laundry list of comments, not an explicit answer.

(1) Is deprecated, but perhaps more because tests of normality can be unreliable, than because of specific worries about error probabilities.

(2) Is often recommended, mainly because many statisticians feel it works better in practice than (1). Specific discussions of what looking at Q-Q plots might do to error probabilities are rare.

(3) Is a comment, not a strategy. Nonparametric tests do not assume normality, but they do have assumptions that need to be considered. Unless data are roughly symmetrical, it is not always clear what they are testing. (Location of population median?)

Something analogous is recommended for the choice of a two-sample test for a difference in normal population means. Unless there is a priori knowledge that group populations have equal variances, it is recommended to use the Welch two-sample t test instead of the pooled two-sample test. The loss in power. using the Welch test (which does not assume equal variances), is usually small compared to the risk of an incorrect decision, using the pooled test.