How close does a distribution have to be to normal, in order for predictions to be accurate?

Question

I know a Kolmogorov-Smirnov test will tell me if a sample distribution belongs to a normal distribution or not, with a certain probability (correct me if I am wrong?).

I performed a KS test on my data and got a p < 2.2e-16, which seems to indicate my distribution is normal?

Below is a histogram of my data with a normal curve overlayed.

If I am to make predictions with this data, can I use normally distributed confidence intervals? My main concern are the columns either side of the mean, that dont seem to fit the distribution.

Answer 1

I know a Kolmogorov-Smirnov test will tell me if a sample distribution belongs to a normal distribution or not

No, it can never tell you a sample was drawn from a normal distribution.

with a certain probability

No, whether or not you reject, it doesn't give you P(normal) nor P(not normal)

I performed a KS test on my data and got a p < 2.2e-16, which seems to indicate my distribution is normal?

No, it tells you your data were not drawn from a normal distribution. (You didn't really need a test to tell you that, I could have told you that for free - and I wouldn't have even looked at your data.)

If I am to make predictions with this data, can I use normally distributed confidence intervals?

No goodness of fit test answers that question.

Unfortunately, neither can I right now, because you haven't give the sort of information required to judge; an answer would depend on a) the things you're doing, b) the properties you care about, and c) your tolerance for deviation from the nominal properties.

For example, it (a) if you're doing a t-interval for a mean, it's not very sensitive to moderate non-normality, but if you're doing an F-interval for a ratio of variances, its coverage properties are quite sensitive to non-normality; it depends on exactly what kind of interval you're talking about.

Answer 2

I think that in your case, the null hypothesis in the KS test does not differ from a normal distribution, so your low p-value indicates that your data is significantly different from a normal distribution. This makes sense because you seem to have a large dataset which makes it easier to reach statistical significance with the test.

However, I don't think that this rules out the possibility of assuming a normal distribution in the context of, for example, a linear regression. In that case I would try the model and check the model using standard tools (plotting fitted values vs residuals etc). I think that in many such cases, assuming normality or not is more of a judgement than an exact science, especially with large sample sizes that will almost always be significantly different than a normal distribution in such tests as KS.