Likert scale survey between two groups

Question

We have tested an educational activity using technology in an undergraduate class where half of the students used it, and the other half completed the traditional paper based activity. We administered a survey at the end of the class to get student perceptions. The survey was not built using constructs, but has statements related to confidence (This educational activity increased my confidence to ....), team work and perceptions on learning.

I want to compare the answers to the survey statements between control and experimental group to see if there are any sig differences. The survey is a 5 point Likert scale. (strongly agree-strongly disagree).

I have used a series of Mann-Whitney tests to compare the medians for each statements. There are about 20 statements.But I am not sure if that is the best way.

Thanks!

Answer 1

This is somewhat addressed in the answer by @BruceET , but the Wilcoxon-Mann-Whitney test is not in general a test of the median. It's a test of stochastic dominance. I think this kind of test makes sense to compare the responses of single Likert items. With implementations of the WMW test that handle ties well, the results are similar to those from ordinal regression, which might be considered the gold-standard for comparing ordinal responses.

As an example, compare the following two vectors of responses. They have the same median, but the WMW test will find a significant difference between the two groups.

A = c(3,3,3,3,3,3,3,3,3,4,4,5,5)
B = c(2,2,2,2,2,2,3,3,3,3,3,3,3)

wilcox.test(A,B)

   ### Wilcoxon rank sum test with continuity correction
   ### 
   ### W = 137.5, p-value = 0.001943

median(A)

   ### 3

median(B)

   ### 3

It might be possible to compare the medians per se, but I don't know that I would recommend Mood's median test for ordinal data like these. I'm not sure.

A = c(3,3,3,3,3,3,3,3,3,4,4,5,5)
B = c(2,2,2,2,2,2,3,3,3,3,3,3,3)
Y = c(A, B)
Group = factor(c(rep("A", length(A)), rep("B", length(B))))
library(coin)

median_test(Y ~ Group,
            distribution = approximate(nresample = 10000))

   ###  Approximative Two-Sample Brown-Mood Median Test
   ###  
   ###  Z = 2.132, p-value = 0.0954

I think plotting the data would be essential. The WMW test treats the responses as ordinal, which makes sense if you want to know if one group has higher responses than the other. But there might be other ways the responses are different. For example, imagine if one group had responses of mostly "3", and the other group had responses equally split between "1" and "5". That would be very interesting, but wouldn't be captured by the WMW test.

A simple bar plot of responses might suffice, ex., rcompanion.org/handbook/images/image061.png , or you might make something prettier, perhaps: i.sstatic.net/lGuxt.png

Answer 2

For small treatment and control groups, you may have difficulty with the Wilcoxon rank sum test of account of the many ties in Likert data.

Consider the following fictitious data with 50 in each group. the ' vectors in sample show different population proportions at the various Likert scores.

set.seed(2022)
x1 = sample(1:5, 50, rep=T, p=c(2,3,3,1,1))
x2 = sample(1:5, 50, rep=T, p=c(1,1,3,3,2))
summary(x1)
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00    2.00    2.50    2.64    3.00    5.00 
summary(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1.0     3.0     3.0     3.5     4.0     5.0 

The resulting sample medians differ, and boxplots show an effect other than a simple shift upward. The samples show populations of different shapes. (The plot for x1 is on the bottom.)

boxplot(x1,x2, horizontal=T, col=c("skyblue2","wheat"))

The Wilcoxon SR test finds a difference between the two samples, but it is probably better to say that the scores in the treatment group 'dominate' those in the contral group rather than that the treatment has shifted the median upward. The implementation of this test in R uses approximate distributions for the ranks, and so reports no difficulties on account of ties.

wilcox.test(x1,x2)

        Wilcoxon rank sum test 
        with continuity correction

data:  x1 and x2
W = 743.5, p-value = 0.0003168
alternative hypothesis: 
 true location shift is not equal to 0

Plots of the empirical CDFs (ECDFs) of the two samples, shows that treatment scores (brown) dominate control scores. That is, the treatment scores tend to be higher and their ECDF plots to the right of the control group scores (thus below).

hdr = "Treatment scores (brown) dominate"
plot(ecdf(x1), col="blue", ylab="ECDF", main=hdr)
 lines(ecdf(x2), col="brown", lty="dotted")

Notes: If your sample sizes are smaller, your implementation of the Wilcoxon RS test cannot handle ties, and you are willing to pretend that Likert scores are numeric, then you might try a Welch two-sample t test (which also shows significance for my fictitious data).

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -3.6499, df = 91.908, p-value = 0.0004354
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -1.3279746 -0.3920254
sample estimates:
mean of x mean of y 
     2.64      3.50 

The implementation of the Wilcoxon SR test in R cannot compute an exact P-value for similar data with sample sizes $n_1=n_2 = 20.$

set.seed(2022)
x1 = sample(1:5, 20, rep=T, p=c(2,3,3,1,1))
x2 = sample(1:5, 20, rep=T, p=c(1,1,3,3,2))

wilcox.test(x1,x2)

        Wilcoxon rank sum test 
        with continuity correction

data:  x1 and x2
W = 112.5, p-value = 0.0154
alternative hypothesis: 
 true location shift is not equal to 0

Warning message:
In wilcox.test.default(x1, x2) : 
 cannot compute exact p-value with ties

t.test(x1, x2)$p.val
[1] 0.02136043