When, if ever, should a normality test be performed on real-world data?

Question

I've recently come across some research studies and internal reports from my company where multiple mean-comparison tests are performed.

The procedure is often as follows: first, the data is checked for normality. If normality is not rejected (with a <5% p-value), a T-test is performed to compare means. Otherwise, non-parametric tests are used instead.

However, from my understanding of Statistics, I see something very wrong about that approach:

First, no real-world data is normal. Small samples are the only reason why normality is not rejected.

Second: it is often the case that, when making the same experiment under slightly different conditions (winter/summer, last year/this year, here/a few km away...), we manage to reject normality only some of the time, resulting in choosing a different testing procedure once we already studied the data. This reminds me a bit of HARKing

Third: if we only care about mean values, we can always perform a T-test. We can get significant differences between populations in a non-parametric test even if they all have the same mean, variance (or any other magnitude of interest)

So, in short: Is this approach legitimate despite all these issues? Under what circumstances? Are those issues real or am I missing something?

Answer 1

There is a marvelous thread on this site: "Is normality testing 'essentially useless'?". It indicates that your take on this matter is essentially correct.

If you have a small sample size then you won't have enough power to detect a true deviation from normality; $p \ge 0.05$ in a normality test doesn't demonstrate normality, it simply doesn't rule it out at that (arbitrary) probability cutoff.

If you have a very large sample size then you typically will rule out normality.

And in either case you've used the data to choose the test, violating the assumptions of the test that you choose.

The one place where you aren't quite right is the statement that "if we only care about mean values, we can always perform a T-test." Yes, you can always calculate a t statistic but the significance test for that t-statistic does assume normally distributed data,* even in tests that allow for different variances between groups. It turns out that t-tests often work well enough when the normality assumption doesn't hold. As another answer to your question notes, transformations of data often can help make data conform better to that assumption.

Also, you don't comment on your company's use of multiple t-tests when there are multiple means to compare. That leads to multiple-comparison issues that might be more troublesome than choosing the tests for significant differences based on an initial normality test.

A Frank Harrell notes in one answer on that first thread, non-parametric tests have reasonably high power even if data are normally distributed, they work well without the normality assumption, and generalize to ordinal cumulative probability models when something more than a 2-sample comparison is needed.

So a more defensible strategy, if you have reason to believe that normality assumptions will be violated so strongly as to pose a problem for t-tests and the like, is just to go with non-parametric tests.

---

*This linked page notes 3 required assumptions: that the sample mean estimate be appropriately normally distributed, that the sample variance estimate be $\chi^2$ with appropriate degrees of freedom, and that these estimates be independent. What it doesn't make immediately clear is that the third assumption is characteristic only of normal distributions. See this answer, linked in a comment by @COOLSerdash, for more details.

Answer 2

I can answer from experience with real-world microbiological data but cannot give an authoritative/general answer. The approach I use and preach involves the following:

1. Graph data as histogram(s) (I know there are issues with this but it's where I start). Does it look skewed? No? Don't rule out t-tests. Yes. Transform it with log, sqrt.
2. If wanting to compare two groups: Calculate variances for both groups and run Levene's test for equal variances on transformed data. Are variances similar (within about 5% of each other)? Yes. Perform t-test. No & sample size is <20? Use non-parametric tests.