2
$\begingroup$

This is a repost from the R forum, as I was told to post here instead.

I would like to test whether there's a significant difference in the mean between this two samples:

withincollaraccuracyknn<-c(0.960, 0.993,0.975,0.967,0.968,0.948)
withincollaraccuracytree<-c(0.953,0.947,0.897,0.943,0.933,0.879)

The data is normally distributed as you can see after running a Shapiro-Wilk test:

> sh<-c(0.960,0.993,0.975,0.967,0.968,0.948,0.953,0.947,0.897,0.943,0.933,0.879)
> shapiro.test(sh)

    Shapiro-Wilk normality test

data:  sh
W = 0.91711, p-value = 0.2628

However, using t.test() or wilcox.test() yield different p-values:

> t.test(withincollaraccuracyknn,withincollaraccuracytree)

    Welch Two Sample t-test

data:  withincollaraccuracyknn and withincollaraccuracytree
t = 3.1336, df = 7.3505, p-value = 0.01552
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.01090532 0.07542802
sample estimates:
mean of x mean of y 
0.9685000 0.9253333 

> wilcox.test(withincollaraccuracyknn,withincollaraccuracytree)

    Wilcoxon rank sum test

data:  withincollaraccuracyknn and withincollaraccuracytree
W = 35, p-value = 0.004329
alternative hypothesis: true location shift is not equal to 0

Could somebody please let me know why? On the Wikipedia page of Mann-Whitney U test, it is stated: "It is nearly as efficient as the t-test on normal distributions".

Note also a Warning when the data is not normally distributed:

> withincollarprecisionknn<-c(0.985,0.995,0.962,1,0.982,0.990)
> withincollarprecisiontree<-c(1,0.889,0.96,0.953,0.926,0.833)
> 
> sh<-c(0.985,0.995,0.962,1,0.982,0.990,1,0.889,0.96,0.953,0.926,0.833)
> 
> shapiro.test(sh)

    Shapiro-Wilk normality test

data:  sh
W = 0.82062, p-value = 0.01623

> 
> 
> wilcox.test(withincollarprecisionknn,withincollarprecisiontree)

    Wilcoxon rank sum test with continuity correction

data:  withincollarprecisionknn and withincollarprecisiontree
W = 30.5, p-value = 0.05424
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(withincollarprecisionknn, withincollarprecisiontree) :
  cannot compute exact p-value with ties

Any help is appreciated. Note that I need to run similar analyses for other datasets having not normally distributed data, so using wilcox.test() instead of t.test() would be an advantage!

$\endgroup$
  • 3
    $\begingroup$ The warning is unrelated to normality of the groups: see the explanation of the p-value calculation on the help page for wilcox.test. Regardless, given that the Wilcoxon test is not comparing means and uses a null hypothesis of equal variances, while the Welch t-test compares means and accounts for unequal variances, it's hard to see why you are comparing the p-values of those tests. $\endgroup$ – whuber May 8 at 17:18
  • $\begingroup$ Even if you propose a situation where the two tests are exactly equally efficient, they will still not reject the same cases. "Equally efficient" is like tossing a pair of coins that have the same bias (e.g. they both come up heads 70% of the time). That doesn't mean they will always come up heads together. $\endgroup$ – Glen_b -Reinstate Monica May 9 at 9:56
  • $\begingroup$ Another thing I'd like to add: You do not need your data to be normal to use the t-test. The t-test is based on the normality of the mean. For larger samples which are sampled iid this is more or less guaranteed to be fulfilled by the CLT. Note however that 6 data points might be too small for the CLT to work. $\endgroup$ – Stefan May 9 at 12:58
  • $\begingroup$ @Stefan Then, can I use Welch t-test for non-normally distributed data (when variances are not equal like this case)? $\endgroup$ – juansalix May 9 at 13:16
  • $\begingroup$ @juansalix you can IF the conditions of the central limit theorem (en.wikipedia.org/wiki/Central_limit_theorem) are fulfilled so that the mean is approximately normal. To check those conditions you'll have to argue three things: 1) your sample size is big enough 2) each observation was obtained independent of the others 3) each observation comes from the same distribution. In your case 2) and 3) depend on the circumstances of data generation and 1) is probably not fulfilled as 6 observations is quite a small sample size. $\endgroup$ – Stefan May 9 at 13:38
12
$\begingroup$

Steady on there!

  1. You have two very small samples there. Statistics is not taught at Hogwarts! No white magic for very small samples.

  2. Not rejecting the null on Shapiro-Wilk doesn't allow the description "is normally distributed", but rather a much more circumspect "not enough evidence to be clear that this isn't normally distributed".

  3. Let's look at graphs, for data separate (left) and data pooled (right).

enter image description here

The graphs would be straight if data were from a normal. I see two things there: Not too bad in terms of (non-)normality for very small samples, but not the same slope, meaning different variability. Checking that, I find the SD for tree is 0.030, and that for knn 0.015: a two-fold difference. The t test should be allowed to follow suit but what you called copes with unequal variability.

  1. Most crucially, no one (competent) promises exactly the same P-values. Different tests focus on different information. For this kind of problem and data, they shouldn't be wildly contradictory, no more, no less.

PS: My own view is that the graph is more interesting and more convincing that any formal test, but those who review your work might want to hear the clank of testing machinery and see the wheels turning.

$\endgroup$
  • $\begingroup$ Well thank you very much for your answer. I'm just wondering then: Should I use wilcox.test() throughout my analysis even if the data is normally distributed? I've a few more datasets just like the one above that I need to compare... Better than t-test()? $\endgroup$ – juansalix May 8 at 17:59
  • 6
    $\begingroup$ Sorry to disappoint, but there can't be a general oracular answer. You say you're interested in testing differences between means. Wilcoxon-Mann-Whitney just does not do that. $\endgroup$ – Nick Cox May 8 at 18:09
  • $\begingroup$ Thank you for your answer. I guess a t-test is the best I can do when the data is normally distributed. However, in case the data is not normally distributed, I can't really use a t-test.. What is the best I can do then to show differences in the mean of the two groups? As you very well said: "those who review your work might want to hear the clank of testing machinery and see the wheels turning." $\endgroup$ – juansalix May 8 at 22:07
  • $\begingroup$ I can't easily extend my answer helpfully. See also @Ben Bolker's helpful answer. An extra dimension is whether your project really breaks down into several little tests, as you seem to imply. Often there is a larger analysis that ties much of the testing together and goes beyond it by fitting an overall model. I think you'd need to ask new questions with more information to move further. In your question you say that your other data are different, now that they are similar! The truth is likely to be a little messy. $\endgroup$ – Nick Cox May 9 at 6:33
  • $\begingroup$ @NickCox at the risk of being too blind or stupid to work it out could you give us the code for producing those charts? I'd like to stea- I mean adapt them. $\endgroup$ – MD-Tech May 9 at 8:12
5
$\begingroup$

I basically agree with @NickCox's answer, here are some further considerations:

  • as pointed out in this answer, if the two groups have different distributions beyond their difference in location, you cannot strictly interpret a significant M-W result as evidence for a change in the mean - you can say that something is different, but it's not necessarily the mean. To modify the linked answer,

if you want to interpret the rejection of the null hypothesis as evidence that [within-collar accuracy for KNN is] greater than [that for trees], then you do need the assumption of equal variance (in fact, equal distributions) between the two populations. If you are satisfied with showing that the distribution of [within-collar accuracy for KNN] differs in some way from that of [within-collar accuracy for trees], then you don't need the extra assumption.

Therefore, in this particular case I would recommend a Welch's t-test (i.e., t-test allowing for different variances in each group), unless the distributions in each group look really weird/far from Normality (I would not use a Shapiro-Wilk test to decide).

More generally, if the distributions appear similar or you're satisfied with showing that the two populations are different (not specifically that the mean differs), I'd recommend Mann-Whitney for its robustness.

If the distributions are quite different and obviously far from Normality and you are specifically interested in testing a difference in location (mean/median/etc.), this becomes a fairly difficult problem ...


  • less importantly, the power loss may be greater than you think: the "$3/\pi \approx 0.95$" efficiency relative to the t-test is an asymptotic result: from Wikipedia,

for large samples from the normal distribution, the efficiency loss compared to the t-test is only 5% ... The relation between efficiency and power in concrete situations isn't trivial ... For small sample sizes one should investigate the power of the Mann–Whitney U test vs the t-test.

(emphasis added)

$\endgroup$
  • $\begingroup$ Thank you very much for pointing that out. Summing everything up with your answer and @NicCox's, should I: 1) Not use a Shapiro-Wilk to test for normality, rather use the graph and see how far that is from a straight slope. 2) In case the graph shows a (reasonably) straight slope and DIFFERENT variances, I can use Welch's test to see if there's a difference in the mean. 3) In case the graph shows NOT normality but similar variance, I can use Mann-Whitney test to see if there's a difference in the mean. 4) In case of normality and equal variance the answer is simple: t-test. Correct? $\endgroup$ – juansalix May 9 at 7:34
  • 1
    $\begingroup$ seems about right $\endgroup$ – Ben Bolker May 9 at 11:36
  • $\begingroup$ i agree also except to underline again that WMW does not test for differences in means. $\endgroup$ – Nick Cox May 9 at 12:07
  • $\begingroup$ @NickCox Thank you for your comment. Hence, what would you say is the best compromise when there's no normality but similar variance and I'm interested in differences in mean? $\endgroup$ – juansalix May 9 at 12:46
  • $\begingroup$ I guess I would bootstrap the difference between means as a summary statistic. $\endgroup$ – Nick Cox May 9 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.