This is a question about a seemingly illogical approach to hypothesis testing and p-value interpretation that I find almost universally endorsed. I’ll cite the answers at Can I trust ANOVA results for a non-normally distributed DV? as an example. Wanting to conduct, say, an ANOVA but faced with non-normal distributions of residuals, people make choices and give rationales for them as if there were just 3 options:
- Decide that the residuals are virtually normal, and treat the p-values of the resulting tests as if they were exact. Cite those values using as many decimal places as desired.
- Decide that normality fits only roughly, but conduct a parametric test as if these data were normal, and again treat the p-values as if they were exact—up to as many decimal places as desired.
- Decide that the normality assumption is not tenable, and abandon the parametric test.
Under this logic, we would have to believe that, as departures from normality become more and more severe, nothing happens to the accuracy of p-values--until some critical threshold, at which point they become completely invalid.
Under a more nuanced view, there whould be another alternative, which is to conduct the parametric test despite some non-normality and treat the resulting p-values as useful approximations, good to perhaps one decimal place. We accept uncertainty and settle for estimates in so many other situations: why not treat p-values the same way?