3
$\begingroup$

I wish to investigate whether endocranial volume (the volume inside the skull) is normally distributed, ideally with a sample size in the hundreds. Few studies have released their data on this subject, but one has. Hoadley and Pearson (1929) gathered data for the endo volume of some 700 ancient Egyptian males. To save space, they reported the data synthetically. "1025" means "1000 to 1050", and so on.

1025 cm3 --> 1
1125 --> 3
1175 --> 10
1225 --> 19
1275 --> 41
1325 --> 77
1375 --> 109
1425 --> 138
1475 --> 118
1525 --> 92
1575 --> 66
1625 --> 28
1675 --> 23
1725 --> 1
1775 --> 2
2025 --> 1

The qqplot looks like this.enter image description here These data appear normal if one outlier is removed at both ends (makes me think of this question) and if they are "jittered" uniformly over 50 cm3:enter image description here

If we ignore the outliers (I don't know what to do of them), this looks fine. Unfortunately, the p-value produced by the Shapiro-Wilk normality test appears to depend a lot on the jittering (i.e., I can't reproduce the p-value). Running the jittering one million times, the p-value is distributed like this:

enter image description here

I therefore cannot report a p-value when the data has been jittered.

What should I do? Is there a way of correcting for continuity before running a Shapiro-Wilk normality test?

$\endgroup$
  • 1
    $\begingroup$ Endocranial volume is not normally distributed; you know this for certain without testing it, since it cannot be negative. Failure to reject with some test simply means your sample size was too small to detect the (possibly small) deviation from normality. If you're tempted to say "I want to know if it's approximately normal, then the usual goodness of fit type hypothesis tests are entirely the wrong tool to answer that question. (What are you trying to find out for? What does that let you do?) $\endgroup$ – Glen_b -Reinstate Monica Jul 22 '17 at 0:14
  • $\begingroup$ @Glen_b Yes Glen, you are of course right, I want to know whether it's approximately normal. People have claimed that Neanderthals from Asia had larger volumes than Neanderthals of Europe. I don't think we have the statistical power to show this, the sample size being too small (3 and 10 respectively). It fails a t-test by a large margin. I'm trying to check whether the assumption of normality of this t-test is verified. If it's normal in Homo sapiens, it's fair to assume it was in Neanderthals. How would you proceed? $\endgroup$ – Pertinax Jul 22 '17 at 0:20
  • $\begingroup$ So to clarify, by "approximately normal" I mean close enough to normal to run a t-test with low sample size. $\endgroup$ – Pertinax Jul 22 '17 at 0:22
3
$\begingroup$

We have here an example of an X-Y problem (asking about an issue with an attempted solution to a previous problem, when it's not actually an ideal way to approach the problem). The solution to an X-Y problem is to address the underlying issue that you started with.

To address the question of whether it's normal or not -- endocranial volume is not normally distributed; we know this for certain without testing it, since it cannot be negative. Failure to reject with some test simply means your sample size was too small to detect the (possibly small) deviation from normality. If you would say "I want to know if it's approximately normal", then the usual goodness of fit type hypothesis tests are entirely the wrong tool to answer that question.

From comments, the underlying question is

I want to know whether it's approximately normal. People have claimed that Neanderthals from Asia had larger volumes than Neanderthals of Europe. I don't think we have the statistical power to show this, the sample size being too small (3 and 10 respectively). It fails a t-test by a large margin. I'm trying to check whether the assumption of normality of this t-test is verified.

You can't "verify" normality. You can reject it or fail to reject it, but you don't have it either way and even if you did you couldn't show it to be the case. But none of this is the issue here.

[See also Is normality testing 'essentially useless'? ... I particularly recommend Harvey M's answer as making several relevant points here]

If it's normal in Homo sapiens, it's fair to assume it was in Neanderthals.

Well, if we insert "approximately" there twice this is probably not very controversial.

by "approximately normal" I mean close enough to normal to run a t-test with low sample size.

Now that's a good question to try to answer, but a Shapiro Wilk test (nor any other goodness of fit test) will simply not answer it. It deals with quite a different question.

In fact it will be likely to lead you to the opposite of the correct conclusion!
It will tend to reject when it matters less for you -- i.e. in large samples (the 700 Egyptian skulls) it will reject normality but your concern here is not whether you can detect non-normality, but a question of how big the effect is on your t-test (sample sizes of 3 and 10 from a distribution that's probably very similar in shape to that, but presumably without the binning).

The question of interest here is whether the (mild) deviation from normality could cause the significance level of the $t$ test could be higher than the level of the test you chose (enough that it would endanger the conclusion - a very small increase of actual significance level would probably be of little concern, but if it were one could adjust significance levels down to compensate). Lower significance level wouldn't seem to be a problem for you.

You want to use the binned Egyptian-skull data as a proxy for the shape of the distribution of the Neanderthal skulls. This seems reasonable (as long as it will be accepted by your audience on subject-matter grounds there's no obvious issue with such an idea).

What you could do

  1. is randomly sample (whether with or without replacement won't matter much but I'll assume its with replacement) sample sizes of 3 and 10 from the Egyptian skull data a very large number of times and see what the proportion of rejections is; this would be an estimate of the actual significance level (you can also get a confidence interval since this is binomial).

    If you're concerned about the discreteness impacting the results (it may have a small effect) you could also (or even instead) smooth the distribution (e.g. via a kernel density estimate) and draw from that KDE (KDE's are very easy to sample from -- it corresponds to adding a little noise / jitter from the kernel to the sampled-with-replacement data). Note that adding noise will increase the variance but we aren't focused on issues of mean or variance here, we're primarily concerned with shape. You want the kernel bandwidth just large enough that the discreteness smooths out -- so it should be on the order of the binwidth, perhaps just a tad larger.

  2. An alternative would be to try some smooth unimodal distributions with similar shape and see how the t-test behaves with those at sample sizes of 10 and 3.

The outcome then (under any of those three options under 1. and 2.) is to see that the significance level is not higher than your chosen level (or at worst only a little larger). Instead of a confidence interval on the significance level as mentioned above, an alternative might be an equivalence/noninferiority type test (where you establish some formal upper limit of what's an acceptable significance level and show that it's below that). I'd be inclined to go the CI route.

However, the next question is which t-test. One could argue that there's likely to be a scale difference rather than a location shift (which might suggest looking at the log-volume instead of the volume but lets leave that aside for now). That would mean that when the means differ, the standard deviations would differ and might lead us to consider a Welch-Satterthwaite type test.

However, I think from what I see here that you're considering two alternatives of the following kind. Under the null you think the two populations are identical (same distribution of endocranial volume). Under the alternative, one group has typically larger skull sizes than the other (which may simply a scale effect rather than a location effect or may impact both).

If that's the case, you don't need a Welch-type test, the ordinary equal-variance test would do just fine, since the situation under the null determines the sample distribution we need to worry about here.

If on the other hand the supposition under the null was they the distributions were not necessarily the same (the spreads might be substantively different even if the null were true), but only that the means didn't differ, then you'd want to use the Welch test. If you're not sure what your null is, take the more general option.

[Yet another alternative would be to consider (in place of a t-test) a permutation test of means on the original data (which doesn't rely on a specific distributional assumption but which would fit the first kind of hypothesis -- the same population distribution under the null). That would remove the need for the consideration of whether the assumptions of the t test hold.]

A challenge: The difficulty with any of this analysis is the extent to which the available Neanderthal skulls can reasonably be treated as a random sample from their respective populations.

[You might also worry about an a priori type power analysis. Here the issue is not so much about non-normality - I'd ignore that because I think it's almost a non-issue by comparison - so much as sample size. If your power to detect a substantial difference was low enough, that would suggest that getting a rejection might be very likely to just be a type I error with a true null. If you have reasonable power to detect what is (before you see the data) a reasonable difference in mean, then this isn't so much of a problem. The kind of alternative you have matters somewhat, though, since we're talking about a false null you might want to consider whether it really would be a scale-shift rather than a location shift]

[What I expect is that investigation would show a very small impact on significance level, likely toward being very slightly conservative because I expect the skull distribution is very slightly skew/heavy-tailed.]

$\endgroup$
  • $\begingroup$ Right on the money! I did the re-sampling with the KDE as you helpfully recommended. Using alpha = 0.01, the proportion of rejections after 100,000 resamplings was of .0088 (CI, .0081 to .0096, with 99% conf.), a slight shift toward conservative, as you predicted. This is a 10% shift, could one correct for it with something like p2 = p*.88? RE random Neanderthal sample: I was wondering the same, especially given that two of the three Asian skulls come from the same cave (Shanidar) and are probably from the same time: there's no way to prove they weren't close kin. $\endgroup$ – Pertinax Jul 22 '17 at 15:07
  • $\begingroup$ RE power: This is a central point of the paper I am writing. With sample sizes so low, power is low, and the relative likelihood of a p-value being spurious is much higher. I indeed wanted to do a power analysis, but one would have to choose an effect size for the alternative hypothesis. I was considering choosing the observed effect size. I believe this would allow one to say something like "if the mean difference in the population were as large as in this sample, the probability of correctly rejecting the null with a random sample of this size would be 12% (alpha=0.01)". $\endgroup$ – Pertinax Jul 22 '17 at 15:22
  • $\begingroup$ Moreover, if the prior on the alternative was low (as I think it should be), say 12%, and one gets a rejection, then it's basically a coin toss if it's a true or false positive. $\endgroup$ – Pertinax Jul 22 '17 at 15:40
  • $\begingroup$ 1. I wouldn't adujst the significance level; I'd simply report (briefly) the analysis that indicates that it's slightly conservative. 2. If you use the observed effect size you're doing a post hoc power analysis. I'd advise against doing that unless you're familiar with the distinctly counterintuitive behavior that can arise if you do that. $\endgroup$ – Glen_b -Reinstate Monica Jul 23 '17 at 12:56
  • $\begingroup$ RE 2. Yes that's what I concluded after experimenting in R. I believe the lower the p-value, the higher the power will appear to be, so effect-size power would be little more than a function of the p-value and hence of little interest. $\endgroup$ – Pertinax Jul 23 '17 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.