I would like to test wether there's a significant difference in the mean between this two samples:

withincollaraccuracyknn<-c(0.960, 0.993,0.975,0.967,0.968,0.948)

The data is normally distributed as you can see after running a Shapiro-Wilk test:

> sh<-c(0.960,0.993,0.975,0.967,0.968,0.948,0.953,0.947,0.897,0.943,0.933,0.879)
> shapiro.test(sh)

    Shapiro-Wilk normality test

data:  sh
W = 0.91711, p-value = 0.2628

However, using t.test() or wilcox.test() yield different p-values:

> t.test(withincollaraccuracyknn,withincollaraccuracytree)

    Welch Two Sample t-test

data:  withincollaraccuracyknn and withincollaraccuracytree
t = 3.1336, df = 7.3505, p-value = 0.01552
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.01090532 0.07542802
sample estimates:
mean of x mean of y 
0.9685000 0.9253333 

> wilcox.test(withincollaraccuracyknn,withincollaraccuracytree)

    Wilcoxon rank sum test

data:  withincollaraccuracyknn and withincollaraccuracytree
W = 35, p-value = 0.004329
alternative hypothesis: true location shift is not equal to 0

Could somebody please let me know why? On the Wikipedia page of Mann-Whitney U test, it is stated: "It is nearly as efficient as the t-test on normal distributions".

Note also a Warning when the data is not normally distributed:

> withincollarprecisionknn<-c(0.985,0.995,0.962,1,0.982,0.990)
> withincollarprecisiontree<-c(1,0.889,0.96,0.953,0.926,0.833)
> sh<-c(0.985,0.995,0.962,1,0.982,0.990,1,0.889,0.96,0.953,0.926,0.833)
> shapiro.test(sh)

    Shapiro-Wilk normality test

data:  sh
W = 0.82062, p-value = 0.01623

> wilcox.test(withincollarprecisionknn,withincollarprecisiontree)

    Wilcoxon rank sum test with continuity correction

data:  withincollarprecisionknn and withincollarprecisiontree
W = 30.5, p-value = 0.05424
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(withincollarprecisionknn, withincollarprecisiontree) :
  cannot compute exact p-value with ties

Any help is appreciated. Note that I need to run similar analysis for other datasets having not normally distributed data, so using wilcox.test() instead of t.test() would be an advantage!

  • $\begingroup$ You are using a different second variable in wilcox.test() compared to t.test(). It has "precision" in the name instead of "accuracy". $\endgroup$ – aosmith May 8 '19 at 16:36
  • $\begingroup$ @aosmith Thanks for pointing that out. Still, the p-value seems to be severely affected by running one or the other test. In case I'm using data that is not normally distributed, should I keep going with wilcox.test() instead? $\endgroup$ – juansalix May 8 '19 at 16:39
  • $\begingroup$ @aosmith Also, please check my edit. $\endgroup$ – juansalix May 8 '19 at 16:45
  • 3
    $\begingroup$ This question is better ask on the Cross Validate site. The two tests are based on different algorithms and each have their pros and cons, so why would you expect the p-values to be the same value? If you cannot assume a normal distribution then using the Wilcox test is a better choice. $\endgroup$ – Dave2e May 8 '19 at 16:56

Browse other questions tagged or ask your own question.