Using wilcox.test() and t.test() in R yielding different p-values

Question

This is a repost from the R forum, as I was told to post here instead.

I would like to test whether there's a significant difference in the mean between this two samples:

withincollaraccuracyknn<-c(0.960, 0.993,0.975,0.967,0.968,0.948)
withincollaraccuracytree<-c(0.953,0.947,0.897,0.943,0.933,0.879)

The data is normally distributed as you can see after running a Shapiro-Wilk test:

> sh<-c(0.960,0.993,0.975,0.967,0.968,0.948,0.953,0.947,0.897,0.943,0.933,0.879)
> shapiro.test(sh)

    Shapiro-Wilk normality test

data:  sh
W = 0.91711, p-value = 0.2628

However, using t.test() or wilcox.test() yield different p-values:

> t.test(withincollaraccuracyknn,withincollaraccuracytree)

    Welch Two Sample t-test

data:  withincollaraccuracyknn and withincollaraccuracytree
t = 3.1336, df = 7.3505, p-value = 0.01552
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.01090532 0.07542802
sample estimates:
mean of x mean of y 
0.9685000 0.9253333 

> wilcox.test(withincollaraccuracyknn,withincollaraccuracytree)

    Wilcoxon rank sum test

data:  withincollaraccuracyknn and withincollaraccuracytree
W = 35, p-value = 0.004329
alternative hypothesis: true location shift is not equal to 0

Could somebody please let me know why? On the Wikipedia page of Mann-Whitney U test, it is stated: "It is nearly as efficient as the t-test on normal distributions".

Note also a Warning when the data is not normally distributed:

> withincollarprecisionknn<-c(0.985,0.995,0.962,1,0.982,0.990)
> withincollarprecisiontree<-c(1,0.889,0.96,0.953,0.926,0.833)
> 
> sh<-c(0.985,0.995,0.962,1,0.982,0.990,1,0.889,0.96,0.953,0.926,0.833)
> 
> shapiro.test(sh)

    Shapiro-Wilk normality test

data:  sh
W = 0.82062, p-value = 0.01623

> 
> 
> wilcox.test(withincollarprecisionknn,withincollarprecisiontree)

    Wilcoxon rank sum test with continuity correction

data:  withincollarprecisionknn and withincollarprecisiontree
W = 30.5, p-value = 0.05424
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(withincollarprecisionknn, withincollarprecisiontree) :
  cannot compute exact p-value with ties

Any help is appreciated. Note that I need to run similar analyses for other datasets having not normally distributed data, so using wilcox.test() instead of t.test() would be an advantage!

Answer 1

Steady on there!

1. You have two very small samples there. Statistics is not taught at Hogwarts! No white magic for very small samples.

2. Not rejecting the null on Shapiro-Wilk doesn't allow the description "is normally distributed", but rather a much more circumspect "not enough evidence to be clear that this isn't normally distributed".

3. Let's look at graphs, for data separate (left) and data pooled (right).

The graphs would be straight if data were from a normal. I see two things there: Not too bad in terms of (non-)normality for very small samples, but not the same slope, meaning different variability. Checking that, I find the SD for tree is 0.030, and that for knn 0.015: a two-fold difference. The t test should be allowed to follow suit but what you called copes with unequal variability.

4. Most crucially, no one (competent) promises exactly the same P-values. Different tests focus on different information. For this kind of problem and data, they shouldn't be wildly contradictory, no more, no less.

PS: My own view is that the graph is more interesting and more convincing that any formal test, but those who review your work might want to hear the clank of testing machinery and see the wheels turning.

Answer 2

I basically agree with @NickCox's answer, here are some further considerations:

• as pointed out in this answer, if the two groups have different distributions beyond their difference in location, you cannot strictly interpret a significant M-W result as evidence for a change in the mean - you can say that something is different, but it's not necessarily the mean. To modify the linked answer,

if you want to interpret the rejection of the null hypothesis as evidence that [within-collar accuracy for KNN is] greater than [that for trees], then you do need the assumption of equal variance (in fact, equal distributions) between the two populations. If you are satisfied with showing that the distribution of [within-collar accuracy for KNN] differs in some way from that of [within-collar accuracy for trees], then you don't need the extra assumption.

Therefore, in this particular case I would recommend a Welch's t-test (i.e., t-test allowing for different variances in each group), unless the distributions in each group look really weird/far from Normality (I would not use a Shapiro-Wilk test to decide).

More generally, if the distributions appear similar or you're satisfied with showing that the two populations are different (not specifically that the mean differs), I'd recommend Mann-Whitney for its robustness.

If the distributions are quite different and obviously far from Normality and you are specifically interested in testing a difference in location (mean/median/etc.), this becomes a fairly difficult problem ...

---

• less importantly, the power loss may be greater than you think: the "$3/\pi \approx 0.95$" efficiency relative to the t-test is an asymptotic result: from Wikipedia,

for large samples from the normal distribution, the efficiency loss compared to the t-test is only 5% ... The relation between efficiency and power in concrete situations isn't trivial ... For small sample sizes one should investigate the power of the Mann–Whitney U test vs the t-test.

(emphasis added)