I have a question about using wilcox.test in R.

I have the following code/data:

a <- c(1,1,1,1,1,2,2,2,2)
b <- c(1,1,1,1,1,1,1,1)
results <- wilcox.test(a,b, conf.int = T, exact = F)

The code above returns the following:

>   Wilcoxon rank sum test with continuity correction
> data:  a and b W = 52, p-value = 0.04271 alternative hypothesis: true
> location shift is not equal to 0 95 percent confidence interval:  0 1
> sample estimates: difference in location 
>           9.415723e-05

As you can see the p-value is less than 0.05 but is not exact. In fact, when I make exact parameter TRUE then I get the warning message that exact p-value cannot be computed:

1: In wilcox.test.default(a, b, conf.int = T, exact = T) :
  cannot compute exact p-value with ties

Here are my questions:

  1. Even though the p-value is less than 0.05, the confidence interval includes 0. How am I supposed to interpret such data? Is the statistical test significant?

  2. After performing two-tailed Wilcoxon test in R, what is the correct way to interpret which group's distribution is significantly higher/lower? Do you look at the confidence interval or the difference in location?

  3. How do you interpret the test result when the confidence interval cannot be computed because all observations are tied?

Thank you.

  • 1
    $\begingroup$ Are the samples from binary distributions? I would hesitate to draw any hard conclusion other than that you should increase the sample size. Based on a power analysis I'd try to at least double it: library(samplesize); n.wilcox.ord(power = 0.8, alpha = 0.05, 0.5, c(5/9, 4/9), c(1, 0)). With such high proportions of equal values between both samples you'd need a quite high n (a quick tests suggest about 10 times what you have) for zero not to be included in the approximate CI. The answer to 3.) is that you really should increase the sample size. $\endgroup$
    – Roland
    Commented Jul 24, 2018 at 7:08
  • $\begingroup$ @Roland: Power calculations given the observed effects are hardly ever useful. If the p value is below or equal the magic limit, then the power is large enough and otherwise it is not ;-). In this case, I'd definitively prefer exact methods (exact Wilcoxon will be equivalent to exact permutation). $\endgroup$
    – Michael M
    Commented Jul 27, 2018 at 18:42
  • 1
    $\begingroup$ @MichaelM On what would you base any power calculation, then, if not on previous information? Are you entirely ruling out the possibility of using previous experimental results to help design future experiments? $\endgroup$
    – whuber
    Commented Jul 27, 2018 at 18:46
  • 1
    $\begingroup$ @whuber: For future experiments it will certainly help but that is a completely different situation. Here, its about one single experiment and my comment refers to that. I should have been more precise. $\endgroup$
    – Michael M
    Commented Jul 27, 2018 at 18:52

1 Answer 1


These data seem wholly unsuited for analysis with a Wilcoxon rank sum test, which assumes continuous data, at least to the extent of avoiding ties. (This Wilcoxon test uses distributions of ranks, which become difficult to compute--even when there are only a few ties. When there are many ties, as in your data, results of the test are essentially meaningless.)

Here is a permutation test of the null hypothesis that the A and B populations are the same against the alternative that the A population has a larger mean than the B population: There are four 2's among 17 observations. All four are found in the first sample. There is no more extreme result. So the probability of this result by chance alone is the one-sided P-value of the test: $$\frac{{9 \choose 4}{8 \choose 0}}{{17 \choose 4}} = 0.0529.$$ Because the P-value exceeds $0.05$ you cannot reject the null hypothesis at the 5% level of significance. [If you want a 2-sided test that the populations differ, as suggested in @whuber's Comment, then the P-value must also include the probability that all four 2's go into Group B: ${9 \choose 4}{8 \choose 0}/{17 \choose 4} + {9 \choose 0}{8 \choose 4}/{17 \choose 4} = 0.0824.]$

With a one-sided P-value so near to 5% you might say there is 'weakly suggestive' evidence that population A has larger values. If this project warrants additional effort, you might try getting more data, as @Rolland has commented.

Note: Often there are so many possible outcomes in a permutation test that there is no simple combinatorial method to find the P-value. Then, one can get a simulated P-value, as in the R code below, where the 'metric' is the difference in means between the two groups. The result with $m = 100,000$ iterations is that the one-sided P-value is just above $0.05$ and the two-sided P-value is just above $0.08$ (both results are in agreement with the combinatorial computations above).

x = c(1,1, 1,1, 1,2, 2,2,2);  y = c(1,1, 1,1, 1,1, 1,1);  all = c(x, y)
set.seed(727);  m = 10^6;  prm.d = numeric(m);  obs.d = mean(x) - mean(y)
for(i in 1:m) {
  prm = sample(all);  prm.d[i] = mean(prm[1:9]) - mean(prm[10:17])  }
mean(prm.d >= obs.d)
[1] 0.053383                    # aprx P-value of one-sided test
mean(abs(prm.d) >= abs(obs.d))
[1] 0.082756                    # aprx P-value of two-sided test

Ref: For an elementary presentation of permutations tests, see Eudey et al. (2010); two-sample tests are discussed in Sect. 3.

  • 2
    $\begingroup$ This is a good explanation, but it doesn't seem quite consistent with the original intention advertised in the software output of conducting a two sided test. $\endgroup$
    – whuber
    Commented Jul 27, 2018 at 16:16
  • 1
    $\begingroup$ @whuber: Just showing the 'best' possible result for these data. Of course, P-value for two-sided test includes probability all 2's are in Group B; result about 0.083. // Will modify Answer to include 2-sided. // BTW, under the delusion that data are 'normal', two-sided Welch t has P-value 0.03527. $\endgroup$
    – BruceET
    Commented Jul 27, 2018 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.