2
$\begingroup$

I have Likert scale data (283 observations) from two groups are I'm trying to interpret the results of a Wilcoxon rank sum test, not being a statistician.

df <- data.frame(
  group = c(FALSE, TRUE, FALSE, TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE),
  value = c(3, 4, 5, 4, 5, 5, 4, 4, 4, 4, 4, 5, 2, 4, 4, 4, 5, 5, 5, 5, 4, 4, 4, 4, 2, 2, 5, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 3, 4, 5, 4, 3, 3, 3, 4, 5, 4, 4, 4, 4, 5, 3, 4, 3, 4, 5, 4, 5, 4, 4, 4, 4, 3, 4, 5, 4, 5, 4, 4, 5, 5, 4, 5, 5, 4, 3, 4, 4, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 3, 3, 4, 5, 4, 4, 4, 4, 4, 4, 4, 5, 5, 3, 4, 2, 3, 3, 3, 4, 4, 4, 4, 4, NA, 5, 4, 3, 5, 4, 4, 4, 4, 4, 3, 4, 3, 5, 4, 4, 4, 5, 5, 4, 4, 4, 4, 4, 3, 3, 5, 5, 5, 4, 4, 4, 3, 4, 4, 4, 4, 4, 2, 4, 4, 5, 4, 4, 4, 4, 5, 4, 4, 5, 4, 4, 5, 4, 4, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 4, 5, 3, 3, 4, 3, 5, 4, 4, 3, 4, 4, 4, 4, 4, 3, 5, 5, 4, 5, 4, 4, 5, 4, 4, 5, 5, 4, 4, 4, 3, 4, 4, 3, 5, 4, 4, 4, 5, 4, 4, 4, 5, 2, 5, 3, 3, 4, 4, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 4, 3, 4, 4, 4, 4, 4, 3, 3, 4, 4, 4, 4, 3, 4, 4, 4, 5, 3, 5, 5, 5, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 3, 4, 5, 5, 4, 3, 5, 5, 4, 2, 4, 3, 3)
)

ggplot(df) +
  geom_jitter(aes(x = group, y = value, color = group, fill = group), width = 0.2, height = 0.2, shape = 21, size = 3) +
  stat_summary(aes(x = group, y = value), fun = "mean", shape = 4, size = 1) +
  scale_color_manual(values = c("#00afbb", "#e7b800")) +
  scale_fill_manual(values = alpha(c("#00afbb", "#e7b800"), 0.4))

enter image description here

> wilcox.test(value ~ group, data = df, conf.int = TRUE)

    Wilcoxon rank sum test with continuity correction

data:  value by group
W = 10719, p-value = 0.04501
alternative hypothesis: true location shift is not equal to 0
95 percent confidence interval:
 -3.977585e-05  6.844053e-05
sample estimates:
difference in location 
          6.195797e-06

First of all I'm surprised to see a significant difference given the seemingly rather similar distributions and the fact that this is a non-parametric test. Given the very small "difference in location" (which seems to correspond to the median difference between samples from both groups), should I conclude that "the difference is significant but negligible"? What also puzzles me is the fact that the confidence interval is centered around 0, I did not expect this given the significant difference.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

What's really going on? Unless you already know what you're looking for, your graphic display is not easy to interpret.

The thing to notice is that the proportion of highest scores (Likert=5) is greater among the False group (about 29%, 51 out of 178), compared with the True group (only about 16%, 18 out of 113).

First, let's look at results of the two-sample Wilcoxon (rank sum test)---skip past the data input:

Your data:

group = c(FALSE, TRUE, FALSE, TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, TRUE, 
      FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, 
      FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, 
      FALSE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, 
      FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, 
      FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, TRUE, TRUE, TRUE, TRUE, 
      FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, 
      FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE, 
      FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, 
      FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, 
      TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, 
      FALSE, TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE, 
      FALSE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, 
      FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, 
      FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, 
      TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, 
      TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, 
      TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, 
      TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE, FALSE, 
      TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, 
      FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, 
      TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE, 
      FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, 
      FALSE, TRUE, TRUE, TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE, 
      FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, 
      FALSE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, FALSE, 
      TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, 
      FALSE, FALSE, FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, 
      FALSE, FALSE, FALSE)

value = c(3, 4, 5, 4, 5, 5, 4, 4, 4, 4, 4, 5, 2, 4, 4, 4, 5, 5, 5, 5, 4, 4, 4,
          4, 2, 2, 5, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 3, 4, 5, 4, 3, 3, 3, 4, 5,
          4, 4, 4, 4, 5, 3, 4, 3, 4, 5, 4, 5, 4, 4, 4, 4, 3, 4, 5, 4, 5, 4, 4,
          5, 5, 4, 5, 5, 4, 3, 4, 4, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 3, 3, 4, 5,
          4, 4, 4, 4, 4, 4, 4, 5, 5, 3, 4, 2, 3, 3, 3, 4, 4, 4, 4, 4, NA, 5, 4,
          3, 5, 4, 4, 4, 4, 4, 3, 4, 3, 5, 4, 4, 4, 5, 5, 4, 4, 4, 4, 4, 3, 3,
          5, 5, 5, 4, 4, 4, 3, 4, 4, 4, 4, 4, 2, 4, 4, 5, 4, 4, 4, 4, 5, 4, 4,
          5, 4, 4, 5, 4, 4, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 4, 5, 3, 3, 4, 3, 5,
          4, 4, 3, 4, 4, 4, 4, 4, 3, 5, 5, 4, 5, 4, 4, 5, 4, 4, 5, 5, 4, 4, 4,
          3, 4, 4, 3, 5, 4, 4, 4, 5, 4, 4, 4, 5, 2, 5, 3, 3, 4, 4, 3, 3, 4, 4,
          4, 4, 4, 4, 4, 4, 3, 3, 4, 3, 4, 4, 4, 4, 4, 3, 3, 4, 4, 4, 4, 3, 4,
          4, 4, 5, 3, 5, 5, 5, 4, 4, 4, 5, 4, 4, 4, 4, 4, 4, 3, 4, 5, 5, 4, 3,
          5, 5, 4, 2, 4, 3, 3)

Wilcoxon RS test finds a difference---but not a difference in medians:

wilcox.test(value ~ group)

        Wilcoxon rank sum test with continuity correction

data:  value by group
W = 10719, p-value = 0.04501
alternative hypothesis: 
  true location shift is not equal to 0

The result (P-value 0.045) is just barely significant at the 5% level. Sometimes, this test is interpreted as a test whether population medians are equal---but not here. Here, the sample medians are the the same (4 for both groups):

summary(value[group==F])
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
  2.000   4.000   4.000   4.065   4.000   5.000       1 
summary(value[group==T])
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2.00    4.00    4.00    3.92    4.00    5.00

So we can say the Wilcoxon test has found that the FALSE group dominates the TRUE group (roughly, tends to have higher values). One way to show this is by looking at the empirical CDF (ECDF) plots for the two groups. The ECDFs use data to imitate the respective population CDFs. The ECDF for FALSE is in blue. Generally, the dominating plot is below and to the right of the other one (requiring higher values to ascend from 0 at left to 1 at right). [Here, the dominating segments are below the others because both groups take only values 2, 3, 4, and 5.]

plot(ecdf(value[group==T]), col="brown", 
     main="ECDFs of FALSE (blue) and TRUE Groups")
  lines(ecdf(value[group==F]), col="blue")

enter image description here

Often when one has to look at ECDF plots to understand the meaning of a Wilcoxon signed rank test, it is difficult to explain to non-statisticians what kind of difference between groups the test has found.

Chi-squared test of contingency table: In order to do a chi-squared test for independence of the group and value variables you begin with a table of counts. The counting is shown in detail below, along with the resulting contingency table TAB.

table(value[group==F])
  2   3   4   5 
  5  20 103  41 

table(value[group==T])
  2  3  4  5 
  2 23 70 18 

TAB=rbind(c(5,20,102,51), c(2,23,70,18))
TAB
     [,1] [,2] [,3] [,4]
[1,]    5   20  102   51
[2,]    2   23   70   18

Here is the chi-squared test resulting from this table.

chisq.out = chisq.test(TAB);  chisq.out

Warning message:
In chisq.test(TAB) : Chi-squared approximation may be incorrect

        Pearson's Chi-squared test

data:  TAB
X-squared = 9.1697, df = 3, p-value = 0.02712

The warning message appears because one of the expected counts is below 5, but only two counts are slightly below 5 and the others are all above 5, so some authors would say the P-value is OK. Here are the expected counts:

chisq.out$exp
         [,1]     [,2]      [,3]     [,4]
[1,] 4.281787 26.30241 105.20962 42.20619
[2,] 2.718213 16.69759  66.79038 26.79381

Furthermore, the implementation of chisq.test in R, permits simulation of a useful P-value even when expected counts are too small. The simulated P-value is about the same as above, significant at the 5% level.

chisq.test(TAB, sim=T)$p.val
[1] 0.02648676

Because there seems to be an association between Group and Value, one can look at Pearson Residuals to find where observed and expected counts are most seriously different. Residuals with largest values call attention to Likert category 5 (4th column of the table).

chisq.out$resi
           [,1]      [,2]       [,3]      [,4]
[1,]  0.3470889 -1.228878 -0.3129148  1.353597
[2,] -0.4356238  1.542337  0.3927326 -1.698870

Ad hoc test of difference of proportions: So it seems worthwhile to look ad hoc at proportions of highest Values between the two Groups.

prop.test(c(51,18),c(178,112))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(51, 18) out of c(178, 112)
X-squared = 5.3266, df = 1, p-value = 0.021
alternative hypothesis: two.sided
95 percent confidence interval:
 0.02346019 0.22814494
sample estimates:
   prop 1    prop 2 
0.2865169 0.1607143

There is a significant difference. So we are back to my first paragraph. This is a difference between the two groups that would be easy to explain to non-statisticians.

Addendum on Stochastic Domination:

Here are data for which it is easier to see that the (dominant) blue ECDF is below and to the right of the brown one. In this example the median of the first sample happens to be larger.

set.seed(2020)
x = sample(1:5, 100, rep=T, p = c(1,1,2,2,4)/10)
y = sample(1:5, 100, rep=T, p = c(2.5,2,2,2,1.5)/10)
wilcox.test(x,y)
       Wilcoxon rank sum test with continuity correction

data:  x and y
W = 6675.5, p-value = 2.742e-05
alternative hypothesis: true location shift is not equal to 0

plot(ecdf(x), col="blue", lwd=2, main="Blue Dominates")
lines(ecdf(y), col="brown", lwd=2, lty="dotted")

enter image description here

table(x)
x
 1  2  3  4  5 
 9  9 23 19 40 
table(y)
y
 1  2  3  4  5 
28 18 13 24 17 
median(x); median(y)
[1] 4
[1] 3

For more technical details see Wikipedia.

Improve this answer
$\endgroup$
1
$\begingroup$

I would like to remind you that Wilcoxon rank-sum (and Mann-Whitney $U$) test is not median test (As far as I know, you need additional assumptions, to test equality of medians using WRS). It ranks the observations from both groups, sums the ranks from one of the groups, and compare it with the expected rank sum. So, it is possible to have identical medians, but still significant test results (I also recommend checking similar questions on CV about the interpretation of WRS/MW, like this one).

I think the main question (...should I conclude that "the difference is significant but negligible"?) is about a limitation of statistical significance, i.e., it might not necessarily mean substantive significance. Now in this case, it is up to you decide on that (it is hard to comment further without knowing what the research is about, how it is conducted, etc.), but you should justify your interpretation, which should be done in reference to the field of research.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.