Can you justify this very common (but seldom discussed) treatment of $p$-values?

Question

This is a question about a seemingly illogical approach to hypothesis testing and p-value interpretation that I find almost universally endorsed. I’ll cite the answers at Can I trust ANOVA results for a non-normally distributed DV? as an example. Wanting to conduct, say, an ANOVA but faced with non-normal distributions of residuals, people make choices and give rationales for them as if there were just 3 options:

1. Decide that the residuals are virtually normal, and treat the p-values of the resulting tests as if they were exact. Cite those values using as many decimal places as desired.
2. Decide that normality fits only roughly, but conduct a parametric test as if these data were normal, and again treat the p-values as if they were exact—up to as many decimal places as desired.
3. Decide that the normality assumption is not tenable, and abandon the parametric test.

Under this logic, we would have to believe that, as departures from normality become more and more severe, nothing happens to the accuracy of p-values--until some critical threshold, at which point they become completely invalid.

Under a more nuanced view, there whould be another alternative, which is to conduct the parametric test despite some non-normality and treat the resulting p-values as useful approximations, good to perhaps one decimal place. We accept uncertainty and settle for estimates in so many other situations: why not treat p-values the same way?

Answer 1

The precision of the p-value is linked to its uncertainty. The uncertainty in the p-value when the test assumptions are not met might reasonable be viewed as part of the general problem of how we incorporate model selection error into our statistical analysis. For more details see

Chris Chatfield, Model Uncertainty, Data Mining and Statistical Inference, Journal of the Royal Statistical Society Series A (Statistics in Society), Vol. 158, No. 3 (1995), pp. 419-466.

I am unaware of any specific guidance on how to address this problem.

Answer 2

I am looking forward to others answers here, but I have one strong suspicion what might be the cause. It is not at all clear in what way the p-value would have to be adjusted since that depends mainly on the way a requirement of the method is adjusted. Note that I don't see any need to restrict the discussion to violations of normality, all that I write here holds for requirements in general.

For example, some violations of the requirements actually lead to a test that is too conservative - meaning the p-value would actually have to be adjusted downwards. One example is a t-test (assuming equal variance) with imbalanced group sizes where the smaller group actually has a smaller variance than the large one.

Then violations can usually not be measured on a numeric scale. Non-normality might mean wide tails, skewness or even bimodality. I don't see any practical way to give a theoretical reason of how much a given p-value has to be adjusted to account for a given violation. I don't even see a set of general guidelines which works in most situations. The situation is complicated by the fact that p-values are already a weird non-linear transformation of effect sizes. So it boils down the fact that it is not really practicable. It should certainly be taken into account when interpreting the result.

This basically covered the practical point of view, but I have also an suspicion that there is a mathematical and theoretical tradition to prefer methods with strong and exact properties when used on data which fulfills the requirements to methods which tend to give more realistic results on real data but which are inferior on the theoretical perfect data. The first kind just lends themselves more to mathematical analysis.