6
$\begingroup$

I have a program generating purportedly normal distributions and I would like to test it. I have a number of issues; perhaps the experts here will help me sort out the essential from the inessential and answer most of these.

  1. I'm looking for a simple test, ideally -- one I can implement without too much trouble.
  2. There may be correlation between adjacent values. Some tests may not be sensitive to this failure if the data are 'otherwise' normally distributed.
  3. Ideally, I'd like to allow a (small!) amount of non-normality. Most tests I've seen allow slightly non-normal data to pass only because a small number of values are tested (where "small" might still mean millions, depending on the size of the deviation). This is reminiscent of this question about the value of normality testing.
Improve this question
$\endgroup$
4
  • $\begingroup$ +1 for an interesting question, particularly because you can tolerate a small amount of non-normality which raises lots of interesting points. But do you have any views on the pros and cons in your particular case of the standard tests - Shapiro-Wilk, Anderson-Darling, etc. - which certainly fit your criterion 1. $\endgroup$
    – Peter Ellis
    Commented Apr 19, 2012 at 6:43
  • $\begingroup$ @PeterEllis: I don't, really. I don't do this often enough to have a good feeling for which would be best. No doubt any of them would pass as long as I didn't use too many data points, but if there was a better way to tolerate a bit of variation than just imprecision I'd like to know. (If there is but it's very complex, I'll go back to not testing too many points -- but at least I'll be wiser.) $\endgroup$
    – Charles
    Commented Apr 19, 2012 at 16:08
  • $\begingroup$ I'm interested to know some more background. E.g., what would be the consequence if your program created distributions that were moderately non-normal as opposed to slightly non-normal? $\endgroup$
    – rolando2
    Commented Apr 19, 2012 at 23:42
  • $\begingroup$ @rolando2: It's a library, so it doesn't do anything with them. I just want to verify that it's correct. But I recognize that, with enough computation, everything is non-normal, and it seemed an interesting problem to distinguish the two. All of this is based on the fact that I recently found a bug in the implementation that caused the numbers to be non-normal -- more than slightly, but not so much that my tests discovered it. $\endgroup$
    – Charles
    Commented Apr 20, 2012 at 2:36

4 Answers 4

Reset to default
6
$\begingroup$

The issues arise with the idea of 'small' amounts of non-normality and 'some' autocorrelation. Until it's clear how to operationalise these then you're stuck with tests of normality (not near normality). There is, as you imply, quite a conceptual difference between an insensitive test of normality and a sensitive test of near normality. You can use the first as the second, but it probably won't be quite right and will behave differently in various limits. It seems to me you can proceed in two ways:

General normality tests do not allow you to control which aspects of non-normality to treat as more serious than others. So can you define what aspect of normality is actually important? If you are more concerned about, e.g. fat tails or skew then you could test for these separately. Similarly, if you estimate the first order autocorrelation you can use the confidence interval on that parameter to determine how much is 'too much'. But you still have to decide what the correct order is (@Jason O. Jensen assumes it is one, but that will depend on the generation process) and whether you trust the test. If I remember correctly, the size of different normality tests (e.g. KS and Shapiro-Wilks) vary with level autocorrelation, sometimes even depending on its sign. And this in addition to the variation in their power with respect to various alternatives...

Second, you say that you generate the data yourself. I'm imagining that either you are testing some kind of random number generator or you are wondering whether something has achieved an asymptotically normal distribution. For the former case, you probably have some idea about what is likely to be wrong, so can test for that, as suggested above. In the latter case, I have less intuition. It is probable that the MCMC convergence literature has something useful to say about this case.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your guess is right -- it's a random number generator. (Sorry, I should have been more explicit!) I'm trying to think of a good general way to handle the situation. I'm reminded of a test of a different part of the system where a function was called at 200 digits of precision and the result checked against a standard correct result. When the function was tweaked to add precision, the last decimal place changed by 1 (making it slightly more accurate) but then the test failed because it didn't match the expected, less correct data! Down the road I'd not like to end up there again. :) $\endgroup$
    – Charles
    Commented Apr 19, 2012 at 16:06
2
$\begingroup$

If point two is your primary concern you could 'lag' the data one observation and then regress the 'raw' data on the 'lagged' data. Do this for a lag each way and decide based on the p value whether the data is sufficiently random.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Welcome to the site! It looks like the original poster's primary concern was with testing for normality, not temporal autocorrelation. In any case, for future reference, this seems more like a comment than an answer (see the FAQ). $\endgroup$
    – Macro
    Commented Apr 19, 2012 at 13:03
  • $\begingroup$ This looks like a sensible approach -- deal with point 2 separately. What kind of test can I use? $\endgroup$
    – Charles
    Commented Apr 19, 2012 at 14:30
  • $\begingroup$ Yes this probably should have been a comment. A rule of thumb is to reject the hypothesis that the two are unrelated if the p-level for the coefficient is less than .05, however, this test is primarily concerned with false positives. You seem to be mostly concerned with a false negative (mistakenly assuming that neighboring values are unrelated). You would have to hypothesize an alternative distribution to test against. looking at the size of the coefficient may likely give a better indication of whether the effect is meaningfully significant. $\endgroup$
    – Jason O. Jensen
    Commented Apr 21, 2012 at 0:45
2
$\begingroup$

Another suggestion would be to compute the Kullback-Leiber divergence or Hellinger distance between your generated data and the normal distribution. That gives you a measure of how non-normal your data is (and hopefully you can determine what a small deviation from normality is).

Improve this answer
$\endgroup$
1
$\begingroup$

The best test that I can think of for near-normality is the visual test in: 

 Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

The vis.test function in the TeachingDemos package for R implements variations on this test. This does assume that you either trust R's random normal generator to be good enough for comparison or that you have another source of normal enough for comparison. This test cannot be automated, but is fairly straight forward and fits the ideas above (and you could find a way to look at the autocorrelation as well if desired).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.