In connection with the question here I came upon a more interesting question. I believe the question is large and distinct enough to have it's own thread. Of course I might be mistaken, in that case I apologize for the clutter...

I have a dataset of experimental values, coming from an unknown distribution. The data is then used in an analysis, and as a measure of significance of the results of the analysis, I would like to sample random data from the same distribution and run my analysis on these datasets (say 1000 times the size of the original dataset) to see if the results from experimental values show significant deviation from randomized data.

I was thinking about drawing samples from a normal distribution, as it feels most natural that the samples come from some normal distribution. However I need to back up my assumption of normality somehow. I was originally thinking of using some sort of a normality test, but after some reading on the matter such as What is 'normality?' and Normality testing: 'Essentially useless?' threads, and of course the Wikipedia article on normality tests; I feel like these tests are not an accepted way of validating normality assumptions.


  1. How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software)
  2. Can a "goodness of fit" score be calculated?

EDIT: As requested, I'll try and add some details about the data. The data at hand are from a biological experiment, however the instrumentation has high variation between the runs. The analysis I've mentioned takes the measured values and using a mathematical model evaluates functional meaning of the measured data. To do so, I need to see how unrelated/uncorrelated, made-up data rates in the same analysis, hence the intent to model by randomized values. I hope this clarifies my point of view.

EDIT2: There has been a series of questions, asking for clarification on the question. Both here and in the comments below I tried to explain my situation to my best ability. It seems like we are suffering from a communication mismatch... I dunno how I can give an example without writing up a long table and complicating things further OR brutally simplifying the bigger picture.

I have no doubt that everyone who took their time and supply a reply has the best intentions, but I really appreciate if you could focus on the question at hand instead of inquiring further and further into the motivations behind why I need to do things this way and not another way.

  • $\begingroup$ Could you elaborate on "using a mathematical model evaluates functional meaning of the measured data" ? How does the mathematical model evaluate the functional meaning? What do you mean by 'evaluate'? What do you mean by 'functional meaning'? $\endgroup$ – onestop Feb 17 '11 at 10:49
  • $\begingroup$ If I understood you want to build up a test of normality that will generate data from a normal random variable (with suitable mean and variance) and compare it to your orginial sample ? Why do you think this can be a better test than usual test of normality ? $\endgroup$ – robin girard Feb 17 '11 at 12:50
  • $\begingroup$ @onestop: this is essentially a systems biology model, we check how different networks are affected by different conditions. I can't really go into detail on what the model is or how it works, as it'll take a long time and place, and I don't think it's very relevant either.. What is important/relevant is; I have abundance levels for a long list of molecules, which come from an instrument with high technical variation, and an unknown underlying distribution. I want to see that my model can distinguish data from real biology, and made-up/random data, from a similar distribution. $\endgroup$ – posdef Feb 17 '11 at 14:28
  • $\begingroup$ @robin girard: not exactly... see the comment above in reply to @onestop. I have a model and I test the significance of the results of my model, by comparing it to data of no biological significance. I intend to sample from normal distribution instead of permuting the same values over and over again. Could this normality approximation be justified somehow, that is my question. $\endgroup$ – posdef Feb 17 '11 at 14:31
  • 2
    $\begingroup$ "just focus on the question at hand instead of inquiring further and further into the motivations behind why". I'm sorry; we're not just being curious or trying to be awkward, honest. Statisticians learn, often the hard way, that it's usually a very bad idea to tell clients how to do something without first enquiring why they want to do it. $\endgroup$ – onestop Feb 17 '11 at 17:26

The question:

  1. How can normality be validated without using visual cues such as QQ plots? (the validation will be a part of larger software)

  2. Can a "goodness of fit" score be calculated?

Although enumerated separately, these parts are (appropriately) one question: you compute an appropriate goodness of fit and use that as a test statistic in a hypothesis test.

Some answers

There are plenty of such tests; the best among them are the Kolmogorov-Smirnov, Shapiro-Wilks, and Anderson-Darling tests. Their properties have been extensively studied. An excellent resource is the work of M. A. Stephens, especially the 1974 article, EDF Statistics for Goodness of Fit and Some Comparisons. Rather than supply a long list of references, I will leave it to you to Google this title: the trail quickly leads to useful information.

One thing I like about Stephens' work, in addition to the comparisons of the properties of various GoF tests, is that it provides clear descriptions of how to compute the statistics and how to compute, or at least approximate, their null distributions. This gives you the option to implement your favorite test yourself. The EDF statistics (empirical distribution function) are easy to compute: they tend to be linear combinations of the order statistics, so all you have to do is sort the data and go. The complications concern (a) computing the coefficients--this used to be a barrier in applying the S-W test, but good approximations now exist--and (b) computing the null distributions. Most of those can be computed or have been adequately tabulated.

What is characteristic about any GoF tests for distributions is that (a) they need a certain amount of data to become powerful (for detecting true deviations) and (b) very quickly thereafter, as you acquire more data, they become so powerful that deviations that are practically inconsequential become statistically significant. (This is very well known and is easily confirmed with simulation or mathematical analysis.) In this is the origin of the reluctance to answer the original question without obtaining substantial clarification. If you have a few hundred values or more, you will find that any of these tests demonstrate your data are not "normal." But does this matter for your intended analysis? We simply cannot say.

  • $\begingroup$ THANK YOU! Not because you answer is the solution to all my problems, but because you have put effort to communicate in a level on which the supplied information can become useful. A lot of times I find myself frustrated and at times borderline angry, when two parties clearly miss one another during exchange of information, and this thread has proven to be one for me. Regarding your suggestions, I'll try to look into the article, but I feel like this has gone on for too long, and in order to finish the software I'm gonna have to implement the analysis as I initially thought... $\endgroup$ – posdef Feb 24 '11 at 9:18
  • $\begingroup$ and leave validation for further development. This way I can hopefully implement something that's acceptable both biologically and statistically (it's surprising how often they don't coincide) $\endgroup$ – posdef Feb 24 '11 at 9:19
  • 1
    $\begingroup$ @posdef Best of luck. I'm sorry to hear about the conflicts of biological and statistical acceptability you have encountered. Any such conflict bothers me, because "statistical unacceptability" is tantamount to unscientific. There has to be a resolution to this. Maybe it needs better communication and understanding on both sides. $\endgroup$ – whuber Feb 24 '11 at 15:11
  • $\begingroup$ Thnx. Regarding the quote: "Any such conflict bothers me, because "statistical unacceptability" is tantamount to unscientific. There has to be a resolution to this. Maybe it needs better communication and understanding on both sides." - I couldn't agree more... $\endgroup$ – posdef Feb 24 '11 at 15:26

If your ultimate aim is, as you say, "to see if the results from experimental values show significant deviation from randomized data" then you'd be better to directly use the observed distribution by performing a permutation (re-randomization) test. This avoids making any assumptions about the 'true' distribution from which the data are drawn.

  • $\begingroup$ true; and I do have permutation based statistics as well. Essentially I am implementing two different ways of analyzing the data, and I need comparable scores (in this case it'll be p-values). The motivation behind my approach is that I have reasons to believe: a) the underlying distribution essentially is very close to normal and, b) the measured values are not large enough to resample as many times, or will add bias to my analysis. But as I said above, there will be permutation based statistics as well. $\endgroup$ – posdef Feb 11 '11 at 10:35
  • 2
    $\begingroup$ @posdef, without a little more information on your analysis, it's a little difficult to give advice. But, unless you are interested in some unusual quantities or have an unusual design, then using a parametric bootstrap in a Gaussian setting is suboptimal. The core of statistical inference is centered around assumptions related to normality (e.g., exponential families) and there is some very powerful optimality theory in these cases. Without good reason, one shouldn't try to avoid it. $\endgroup$ – cardinal Feb 11 '11 at 13:51
  • $\begingroup$ @cardinal: can't really say I follow your reasoning, or rather, I am not sure I fully understand what you are saying. :) But I can try and explain my analysis in further detail, if necessary. $\endgroup$ – posdef Feb 11 '11 at 15:31
  • $\begingroup$ @posdef, any additional detail you could provide would be both helpful and interesting. My point was that if you make the leap to assuming normality, it's unlikely that a parametric bootstrap would be the desired approach to go about conducting your inference. But, maybe your situation is complex/unique enough that it does have a role to play. $\endgroup$ – cardinal Feb 11 '11 at 16:10
  • $\begingroup$ @cardinal: just added some details, regarding the nature of the data, and analysis, followed with my motivations for the analysis I proposed. $\endgroup$ – posdef Feb 14 '11 at 9:23

The best argument I have heard about using the normal distribution is one of "sufficient statistics" given by Larry Bretthorst in his PhD thesis on spectral estimation.

Basically because the normal distribution has a set of sufficient statistics (mean and covariance matrix), then this is the only thing that matters when fitting your data. So if the "true" distribution is not normal, but it gives you the same set of sufficient statistics, then you will get exactly the same answer as if it was normal, whenever the parameters are linear in these sufficient statistics (if non-linear, then its a projection onto the error space). This is in line with "BLUE" type estimators.

So in this sense, testing for normality is "stupid". But checking the residuals is certainly a good idea. This is because it can help improve your model, by looking for systematic patterns in them. The normality tests are usually designed to pick up these systematic patterns, so in this sense they are good. The problem with them, is that they don't "give you a clue" about how to fix the "non-normality". Looking at the residuals does, though. e.g. see curvature in residuals: put a quadratic term in, see "harmonic" behaviour: put a sine or cosine function in, etc.

Another justification comes from the MaxEnt world. The normal distribution is the one with the largest entropy (i.e. uncertainty) among all continuous distributions on the reals with given mean and variance. So this means that the normal "assumes the least", in a sense for a given mean and variance. This means that "Nature" has to work incredibly hard to move to a part of the "distribution space" which is non-normal and keep the mean and variance fixed. The normal can be realised in a massively larger number of ways compared to any other with fixed mean and fixed variance. So a departure usually means there is some "unspecified" constraint operating on the data - which could be thought as a "covariate missed".

It is because the normal distribution is non-robust that it allows you to "learn from your mistakes". If you are wrong, the normal distribution will pick this up by producing large residuals. When you use "robust" or "non-parametric" methods, it becomes more difficult to see where a model can be improved.

And as a final note: it matters not one iota if the normal distribution is an awful fit, unless you have something to replace it with!


I would maybe (I don't know if it is feasible in your context) suggest another approach. You could force your experimental data to follow a standard normal distribution by applying a normal quantile tranformation on it. The principle is to

1) rank your values from high to low

2) assign the value of r-th rank the (r-0.5)/n th quantile of the standard normal distribution. This ensures that your data is N(0,1)

3) perform the analysis on the transformed data

I feel that if your original data is close to normal this would not change much your inference. Then of course you can simulate data from a standard normal distribution and it will fit your experimental data.

  • $\begingroup$ @user3386 This is a nice idea. But the proposed analysis will draw many values from a normal distribution. When the quantile of such a value exceeds (n-1/2)/n or is less than (1/2)/n, what data value would correspond to it? There is no good answer to this with the information we have. In short, extrapolation--which appears to be an essential need--is impossible with your proposal. $\endgroup$ – whuber Feb 23 '11 at 20:51
  • $\begingroup$ @user3386 - this is not a very good idea if you want to know what's going on with your data. It is for the data to tell you whether it is or isn't "normal". The only time this would be a good idea is when it is useless - because the normal approximation already holds. It's like you're "telling nature how its gona be". $\endgroup$ – probabilityislogic Feb 24 '11 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.