Justifying normal approximation of experimental data

Question

Background:

In connection with the question here I came upon a more interesting question. I believe the question is large and distinct enough to have it's own thread. Of course I might be mistaken, in that case I apologize for the clutter...

I have a dataset of experimental values, coming from an unknown distribution. The data is then used in an analysis, and as a measure of significance of the results of the analysis, I would like to sample random data from the same distribution and run my analysis on these datasets (say 1000 times the size of the original dataset) to see if the results from experimental values show significant deviation from randomized data.

I was thinking about drawing samples from a normal distribution, as it feels most natural that the samples come from some normal distribution. However I need to back up my assumption of normality somehow. I was originally thinking of using some sort of a normality test, but after some reading on the matter such as What is 'normality?' and Normality testing: 'Essentially useless?' threads, and of course the Wikipedia article on normality tests; I feel like these tests are not an accepted way of validating normality assumptions.

Question:

1. How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software)
2. Can a "goodness of fit" score be calculated?

---

EDIT: As requested, I'll try and add some details about the data. The data at hand are from a biological experiment, however the instrumentation has high variation between the runs. The analysis I've mentioned takes the measured values and using a mathematical model evaluates functional meaning of the measured data. To do so, I need to see how unrelated/uncorrelated, made-up data rates in the same analysis, hence the intent to model by randomized values. I hope this clarifies my point of view.

---

EDIT2: There has been a series of questions, asking for clarification on the question. Both here and in the comments below I tried to explain my situation to my best ability. It seems like we are suffering from a communication mismatch... I dunno how I can give an example without writing up a long table and complicating things further OR brutally simplifying the bigger picture.

I have no doubt that everyone who took their time and supply a reply has the best intentions, but I really appreciate if you could focus on the question at hand instead of inquiring further and further into the motivations behind why I need to do things this way and not another way.

Answer 1

The question:

1. How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software)

2. Can a "goodness of fit" score be calculated?

Although enumerated separately, these parts are (appropriately) one question: you compute an appropriate goodness of fit and use that as a test statistic in a hypothesis test.

Some answers

There are plenty of such tests; the best among them are the Kolmogorov-Smirnov, Shapiro-Wilks, and Anderson-Darling tests. Their properties have been extensively studied. An excellent resource is the work of M. A. Stephens, especially the 1974 article, EDF Statistics for Goodness of Fit and Some Comparisons. Rather than supply a long list of references, I will leave it to you to Google this title: the trail quickly leads to useful information.

One thing I like about Stephens' work, in addition to the comparisons of the properties of various GoF tests, is that it provides clear descriptions of how to compute the statistics and how to compute, or at least approximate, their null distributions. This gives you the option to implement your favorite test yourself. The EDF statistics (empirical distribution function) are easy to compute: they tend to be linear combinations of the order statistics, so all you have to do is sort the data and go. The complications concern (a) computing the coefficients--this used to be a barrier in applying the S-W test, but good approximations now exist--and (b) computing the null distributions. Most of those can be computed or have been adequately tabulated.

What is characteristic about any GoF tests for distributions is that (a) they need a certain amount of data to become powerful (for detecting true deviations) and (b) very quickly thereafter, as you acquire more data, they become so powerful that deviations that are practically inconsequential become statistically significant. (This is very well known and is easily confirmed with simulation or mathematical analysis.) In this is the origin of the reluctance to answer the original question without obtaining substantial clarification. If you have a few hundred values or more, you will find that any of these tests demonstrate your data are not "normal." But does this matter for your intended analysis? We simply cannot say.

Answer 2

If your ultimate aim is, as you say, "to see if the results from experimental values show significant deviation from randomized data" then you'd be better to directly use the observed distribution by performing a permutation (re-randomization) test. This avoids making any assumptions about the 'true' distribution from which the data are drawn.

Answer 3

The best argument I have heard about using the normal distribution is one of "sufficient statistics" given by Larry Bretthorst in his PhD thesis on spectral estimation.

Basically because the normal distribution has a set of sufficient statistics (mean and covariance matrix), then this is the only thing that matters when fitting your data. So if the "true" distribution is not normal, but it gives you the same set of sufficient statistics, then you will get exactly the same answer as if it was normal, whenever the parameters are linear in these sufficient statistics (if non-linear, then its a projection onto the error space). This is in line with "BLUE" type estimators.

So in this sense, testing for normality is "stupid". But checking the residuals is certainly a good idea. This is because it can help improve your model, by looking for systematic patterns in them. The normality tests are usually designed to pick up these systematic patterns, so in this sense they are good. The problem with them, is that they don't "give you a clue" about how to fix the "non-normality". Looking at the residuals does, though. e.g. see curvature in residuals: put a quadratic term in, see "harmonic" behaviour: put a sine or cosine function in, etc.

Another justification comes from the MaxEnt world. The normal distribution is the one with the largest entropy (i.e. uncertainty) among all continuous distributions on the reals with given mean and variance. So this means that the normal "assumes the least", in a sense for a given mean and variance. This means that "Nature" has to work incredibly hard to move to a part of the "distribution space" which is non-normal and keep the mean and variance fixed. The normal can be realised in a massively larger number of ways compared to any other with fixed mean and fixed variance. So a departure usually means there is some "unspecified" constraint operating on the data - which could be thought as a "covariate missed".

It is because the normal distribution is non-robust that it allows you to "learn from your mistakes". If you are wrong, the normal distribution will pick this up by producing large residuals. When you use "robust" or "non-parametric" methods, it becomes more difficult to see where a model can be improved.

And as a final note: it matters not one iota if the normal distribution is an awful fit, unless you have something to replace it with!

Answer 4

I would maybe (I don't know if it is feasible in your context) suggest another approach. You could force your experimental data to follow a standard normal distribution by applying a normal quantile tranformation on it. The principle is to

1) rank your values from high to low

2) assign the value of r-th rank the (r-0.5)/n th quantile of the standard normal distribution. This ensures that your data is N(0,1)

3) perform the analysis on the transformed data

I feel that if your original data is close to normal this would not change much your inference. Then of course you can simulate data from a standard normal distribution and it will fit your experimental data.