# How do I choose the best statistical test for analyzing differences between groups in survey data?

I have n=1000 observations and need to compare 2 demographic groups to the rest of the data (any observations not in those groups; so, 3 groups) to answer whether or not there are significant differences between those groups' answers on 10 separate survey questions. Each question has an identical 5-point scale, from Strongly Agree(5) to Strongly Disagree(1).

I'm struggling with how to set this up. I considered creating 3 bins for the different groups, then running comparisons between them for each question. Null is there are no differences.

Also considered just splitting the 2 main groups out one at a time and comparing to the rest of the data (which would include the other group).

I'm not sure if this is right, or what test I should be running. Any help appreciated.

Suppose you have Likert data for three groups. If you are willing to treat the data as interval numerical data, you might use a one-way ANOVA to make an initial determination whether there are any statistically differences at all among the three groups. If there are differences, then do ad hoc Welch t tests to see which groups are different. (Not everyone agrees that Likert data should be treated as interval data.)

If you want to regard the data as ordinal (a non-controversial point of view), then you could use a Kruskal-Wallis test to look for significant differences among the three groups, and follow up with ad hoc two-sample Wilcoxon tests as appropriate.

Simulated data. Here are simulated data for three groups, and examples of how you might run the tests. [Everything below is done in R.]

set.seed(616)
x1 = sample(1:5, 250, rep=T, p=c(1,3,3,2,1))
x2 = sample(1:5, 250, rep=T, p=c(1,2,3,2,1))
x3 = sample(1:5, 250, rep=T, p=c(1,2,2,3,3))
x = c(x1,x2,x3);  g=rep(1:3, each=250)

summary(x1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.000   2.000   3.000   2.828   4.000   5.000
summary(x2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.00    2.00    3.00    3.04    4.00    5.00
summary(x3)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.000   3.000   4.000   3.548   5.000   5.000

boxplot(x ~ g, col="skyblue2", notch=T)


Notches in the boxplots are nonparametric interval estimates of medians, calibrated for roughly comparing two groups at a time. Nonoverlapping notches suggest differences in location. So maybe groups 1 and 2 are the same and group 3 is different from both. Formal tests below support this pattern of differences.

ANOVA and ad hoc t tests for interval data. A one-way ANOVA that does not assume equal variances among groups finds highly significant differences among groups.

oneway.test(x ~ g)

One-way analysis of means
(not assuming equal variances)

data:  x and g
F = 22.931, num df = 2.00, denom df = 497.12,
p-value = 2.979e-10


Welch two-sample t tests show highly significant differences between group 3 and groups 1 and 2. For an ad hoc test I would not consider the 4% P-value as significant for an ad hoc test. (Using Bonferroni protection against false discovery I would want to see a P-value below about 1.6% for that.)

t.test(x1,x2)$$p.val [1] 0.03990473 t.test(x1,x3)$$p.val
[1] 7.73433e-11
t.test(x2,x3)$p.val [1] 3.09803e-06  For the t tests, I have shown only the P-values. You might want to remove the $p.val from the code to see the complete printouts.

For ordinal data: Kruskal-Wallis test for three groups; Wilcoxon ad hoc tests. A Kruskal-Wallis nonparametric test finds highly significant differences among groups.

kruskal.test(x ~ g)

Kruskal-Wallis rank sum test

data:  x by g
Kruskal-Wallis chi-squared = 45.983, df = 2,
p-value = 1.035e-10


Two-sample Wilcoxon (rank sum) tests find the same pattern of differences that we saw above with the t tests.

wilcox.test(x1,x2)$$p.val [1] 0.02118815 wilcox.test(x1,x3)$$p.val
[1] 1.278709e-10
wilcox.test(x2,x3)\$p.val
[1] 2.276063e-06


Note: It is possible to find significant differences among three groups and then not to be able to resolve completely the true pattern of differences. With your real data, you might be able to see that groups 1 and 3 differ significantly, but not be able to determine that group 2 (with intermediate values) is significantly different from either.

• wonderful answer, thanks! I was on the right track with ANOVA and K-W, then. I planned to do both and compare. Further, when comparing the groups using Welch or Wilcox, I'm interested in knowing whether the groups are different (and if greater/less than the other). So would I use the default two-tail method? And how do you tell which group is greater or less than the other in the output? Also, is it correct to go through this entire procedure for each of the 10 questions for which I have data from the 3 groups? – telemachus Jun 17 '20 at 20:42
• It's not good practice to shop around among several potentially valid tests among the three groups, Pick standard ANOVA, 'Welch-Satterthwaite' ANOVA (not assuming equal variances) or K-W, whichever has assumptions best matching your data. // Do two-sided tests unless you have strong reason for one-sided. For a 2-sided test, look at sample mean/medians to detect the observed direction of difference. // Whether you look at each of 10 questions separately or at averages depends on your interests and objectives. Are questions closely related or on scattered topics? Maybe avg if related. – BruceET Jun 17 '20 at 21:01
• They're somewhat related, but doing separate analyses on each seems safer. //I think I get it: using your example, for a two-sided test, I'd look at the summary output, see that mean x3>x2, so the t-test shows x3 is significantly greater, based on p=alpha/2? Thanks for the patience. – telemachus Jun 17 '20 at 21:22