I have a set of measurements coming from a manufacturing processes. I want to test if the measurements come from a normal distribution. If I understadn correctly, it's wrong to use K-S test or A-D test with mean and variance estimated by the sample. BTW, that's what everybody else in my team is doing. Of course, that doesn't make it right! It's just that I cannot rely on my coworkers for guidance on this topic. However, intuitively I suppose that, having fitted the parameters, now the test has much less power to reject the null. Now, even with estimated mean and variance, I usually get absurdly low p-values (stuff like 0.0001 or less!). Thus, I think I'm safe to say that my data are definitely not normal. Is that correct?
I know that the right way to approach this issue would be to follow the procedure in
but I don't understand that procedure. I will write another question about it.
EDIT: people asked why I'm doing the normality test. Usually in my company they're done because when you investigate for variation sources in a MFG process, you use different tests whether subgroups are normally distributed or not. For example, normally distributed subgroups => use F-test to assess Homogeneity of Variance. Non normally distributed => use Levene's test. I guess that's related to the assumptions of each test.
My case is different. I need to perform an uncertainty quantification study of the performance of a machine: I want to propagate the manufacturing process variability across the chain of codes which are used to compute the performance of the machine, in order to estimate the uncertainty in the machine performance. I thus need to perform a Monte Carlo analysis (or Polynomial Chaos, or Gaussian Process - pick your favourite tool). In any case, I need to be able to sample from a distribution, and I was demanded by management to "keep things simple!". Because of this common "test for normality" mentality, in people minds "keep things simple!" means "just use normal distributions for inputs, and don't waste any precious minutes of work on mumbo-jumbo stuff!".
Now, of course MFG data can't have perfectly normal deviations - a diameter must be positive, an angle has bounded range, etc. But that's not the real issue here: for example, my diameter data have such a small coefficient of variation that a Gaussian distributon and a truncated Gaussian with lower limit 0 would look basically the same: P(D<0|normal distribution)= 1e-100 or something like that. Thus, if the only issue was that MFG data are bounded data, a Gaussian pdf could still be a reasonable proxy. However, I see much bigger deviations from normality: p=0.0001 or less. Thus I need the normality test as an objective measure to prove to my managers that, by using normal distributions for inputs, my uncertainty quantification study would be completely messed up.