16
$\begingroup$

In R, I have a sample of 348 measures, and want to know if I can assume it is normally distributed for future tests.

Essentially following another Stack answer, I am looking at the density plot and the QQ plot with:

plot(density(Clinical$cancer_age))

enter image description here

qqnorm(Clinical$cancer_age);qqline(Clinical$cancer_age, col = 2)

enter image description here

I do not have a strong experience in Statistics, but they look like examples of normal distributions I have seen.

Then I am running the Shapiro-Wilk test:

shapiro.test(Clinical$cancer_age)

> Shapiro-Wilk normality test

data:  Clinical$cancer_age
W = 0.98775, p-value = 0.004952

If I interpret it correctly, it tells me it is safe to reject the null hypothesis, which is that the distribution is normal.

However, I have encountered two Stack posts (here, and here), which strongly undermine the usefulness of this test. Looks like if the sample is big (is 348 considered as big?), it will always say that the distribution is not normal.

How should I interpret all that? Should I stick with the QQ plot and assume my distribution is normal?

Improve this question
$\endgroup$
10
  • 5
    $\begingroup$ The q-q plot seems to show a departure from normal in the tails. Also any useful test of goodness of fit will reject in very large samples simply because there will be small departures from normality that are detected.. It is not a criticism of the Shapiro - Wilk test but rather a feature of testing for goodness of fit. $\endgroup$
    – Michael R. Chernick
    Jun 7, 2017 at 13:47
  • 4
    $\begingroup$ Why is assuming a normal distribution important to you? What do you intend to do based on that assumption? $\endgroup$
    – Roland
    Jun 7, 2017 at 13:47
  • 6
    $\begingroup$ Just to add to Roland's comments- many tests that formally assume a normal distribution are actually fairly robust under slight departures from normality (e.g. because the test statistic's distribution is asymptotically correct). If you can elaborate about what you intend to do you may get more helpful answers. $\endgroup$
    – P.Windridge
    Jun 7, 2017 at 13:56
  • 1
    $\begingroup$ @mdewey, sharp observation! It is not age at incidence, but the tumor's "age" measured by DNA methylation. $\endgroup$
    – francoiskroll
    Jun 7, 2017 at 14:01
  • 2
    $\begingroup$ I think it would be worth examining the small number of extreme observations just to check if they are measurement errors. $\endgroup$
    – mdewey
    Jun 7, 2017 at 14:04

2 Answers 2

Reset to default
14
$\begingroup$

You do not have a problem here. Your data my be slightly non-normal, but it is normal enough that it shouldn't pose any problems. Many researchers do statistical tests assuming normality with far less normal data than those that you have.

I would trust your eyes. The density and Q-Q plots look reasonable, despite some slight positive skew on the tails. In my opinion, you do not need to worry about non-normality for these data.

You have an N of about 350, and p-values are very dependent on sample sizes. With a large sample, almost anything can be significant. This has been discussed here.

There are some incredible answers on this very popular post that basically comes to the conclusion that conducting a null-hypothesis significance test for non-normality is "essentially useless." The accepted answer on that post is a fabulous demonstration that, even when data were generated from a nearly Gaussian process, a high enough sample size makes the non-normal test significant.

Sorry, I realized that I linked to a post you had mentioned in your original question. My conclusion still stands, though: Your data are not so non-normal that it should pose problems.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Just because some.researchers are very sloppy doesn't mean you can be a bit sloppy :). However I agree with that many statistical tests that formally assume normality are actually fairly tolerant of what you feed the $\endgroup$
    – P.Windridge
    Jun 7, 2017 at 13:59
  • 2
    $\begingroup$ "Just because some.researchers are very sloppy doesn't mean you can be a bit sloppy :)" Fair point; that was a bad argument on my part. "However I agree with that many statistical tests that formally assume normality are actually fairly tolerant of what you feed them." Yes, indeed. Any quant professor I've had has looked at Q-Q plots like this and said, "Yeah, that's OK." $\endgroup$
    – Mark White
    Jun 7, 2017 at 14:01
6
$\begingroup$

Your distribution is not normal. Look at the tails (or lack thereof). Below is what you would expect from a normal QQ plot.

enter image description here

Refer to this post on how to interpret various QQ plots.

Keep in mind that while a distribution may not technically be normal, it may be normal enough to qualify for algorithms which require normalcy.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ What are you talking about, I ran 9 normal qq plots form samples directly form a normal distribution using the code set.seed(100) par(mfrow = c(3,3)) for(i in 1:9){ x <- rnorm(350) qqnorm(x) qqline(x) } and plot (3,2) look very similar to OP's situation. $\endgroup$
    – Josh
    Jun 8, 2017 at 7:13
  • 1
    $\begingroup$ Typically, you don't want to focus on the tails because they will often be weird, although extremely bad tails will give you poor results. You really want to focus on the middle. $\endgroup$
    – Josh
    Jun 8, 2017 at 7:17
  • $\begingroup$ you are incorrect Josh. please appeal to a normal test to check if the null hypothesis of normalcy is rejected. $\endgroup$
    – redress
    Jun 8, 2017 at 13:46
  • 1
    $\begingroup$ You're right. I initially read your post as the qq plots were not normal enough, and I apologize. $\endgroup$
    – Josh
    Jun 8, 2017 at 14:28
  • 2
    $\begingroup$ @Josh, the middle of the distribution hardly matters for hypothesis tests; it's the tails that matter. You have that backwards. $\endgroup$
    – gung - Reinstate Monica
    Jun 8, 2017 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.