I've recently come across some research studies and internal reports from my company where multiple mean-comparison tests are performed.
The procedure is often as follows: first, the data is checked for normality. If normality is not rejected (with a <5% p-value), a T-test is performed to compare means. Otherwise, non-parametric tests are used instead.
However, from my understanding of Statistics, I see something very wrong about that approach:
First, no real-world data is normal. Small samples are the only reason why normality is not rejected.
Second: it is often the case that, when making the same experiment under slightly different conditions (winter/summer, last year/this year, here/a few km away...), we manage to reject normality only some of the time, resulting in choosing a different testing procedure once we already studied the data. This reminds me a bit of HARKing
Third: if we only care about mean values, we can always perform a T-test. We can get significant differences between populations in a non-parametric test even if they all have the same mean, variance (or any other magnitude of interest)
So, in short: Is this approach legitimate despite all these issues? Under what circumstances? Are those issues real or am I missing something?