# Investigating Group Membership's Impact on Questionnaire Scores: Need Assistance with Kruskal-Wallis Test and Data Analysis

I'm currently working on a research project where I want to explore whether there is a relationship between group membership and scores obtained on a questionnaire. I have three distinct groups, and the questionnaire scores follow a continuous distribution without being normally distributed. After some research, I decided to use the Kruskal-Wallis test to analyze the data.

Here's the R code I used for the Kruskal-Wallis test, with the result:

kruskal.test(Score, Group, data = Data)
# Kruskal-Wallis chi-squared = 24.612, df = 2, p-value = 4.524e-06


The result is very much significant, implying that belonging to a group can affect the score. However, when I tried to visualize the data and explore descriptive statistics, I noticed that the groups' distributions seem to overlap quite a bit.

Data %>%
group_by(Group) %>%
summarise(
count = n(),
mean = mean(Score),
sd = sd(Score),
median = median(Score),
IQR = IQR(Score)
)


Here are the summary statistics for each group:

| Group    | count | mean    | sd      | median | IQR |
|----------|-------|---------|---------|--------|-----|
| Grp1     | 367   | 41.335  | 4.466   | 42.0   | 5.0 |
| Grp2     | 337   | 42.745  | 3.821   | 43.5   | 4.0 |
| Grp3     | 356   | 42.702  | 3.720   | 43.0   | 4.5 |


As a beginner in both R and statistics, I'm really unsure about the discrepancy in results I'm seeing. I've been trying to figure out if I made any mistakes during the analysis, but I'm hitting a roadblock. I would be extremely grateful if someone could offer their expertise and guide me through the process.

Any advice, suggestions, or insights on what might be happening here would be immensely helpful!

I tried both forms of the kruskal.test() (formula and x,g) and I tried removing the outliers.


lower_bound <- Q1 - 1.5 * IQR
upper_bound <- Q3 + 1.5 * IQR

Data_no_outliers <- Data %>%
filter(Score >= lower_bound, Score <= upper_bound)

summary(Data_no_outliers\$Score)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
32.50   40.50   43.00   42.63   45.00   49.00

kruskal.test(Score ~ Group, data = Data_no_outliers)
# Kruskal-Wallis chi-squared = 18.777, df = 2, p-value = 8.37e-05

• Have you tried displaying boxplots of scores against 3 groups to find some clues about trends? You are using KW but seem to have Gaussian distribution, is there a rationale for this? KW is less powerful than a parametric test like T-test or Welch-correction of T-test. I don't understand why you want to exclude outliers since they are part of the real distribution, if you remove them, then you assume they are abberant values. Outliers=/=Abberant values Commented Jul 21, 2023 at 8:42
• Additionally, for large samples statistical tests will detect small differences between your groups. Here, you have a lot of data in each group so the odds of getting a significant p-value are large even though the difference is very small. Commented Jul 21, 2023 at 9:41
• Thanks! The 'Score' represents the total questionnaire score. Upon plotting, I observed a left-skewed distribution. To check normality, I used the Shapiro-Wilk test, yielding an extremely small p-value. This led me to consider non-parametric tests, which seemed appropriate at the time. The boxplots and table display overlapping boxes for different groups, making me doubt my (correct) use of KW. I tried removing outliers to diagnose the issue, but it didn't provide any helpful insights. Any suggestions on how to proceed would be greatly appreciated!
– M93847
Commented Jul 21, 2023 at 10:12
• Your first line is kruskal.test(Score, Group, data = Data), with a comma instead of a tilde (~) for formula notation. If this notation does not raise an error like object 'Score' not found (similar for "Group") you have some objects "Score" and "Group" hanging around in your search path ("workspace") which might differ frome the same-named columns of your dataframe Data which you intend to test.
– I_O
Commented Jul 21, 2023 at 10:41
• @PeterFlom: the difference is where it goes looking for names. Try kruskal.test(Sepal.Length ~ Species, data = iris) vs. kruskal.test(Sepal.Length, Species, data = iris).
– I_O
Commented Jul 21, 2023 at 14:23

Note what kruskal.test does. From R help:

kruskal.test performs a Kruskal-Wallis rank sum test of the null that the location parameters of the distribution of x are the same in each group (sample). The alternative is that they differ in at least one.

From Wikipedia:

A significant Kruskal–Wallis test indicates that at least one sample stochastically dominates one other sample.

Essentially, this would mean that the ordered scores for one group are higher than the other, but not how much higher (because it uses ranks). E.g.

set.seed(1234)

x <- rnorm(1000, 10, 1) #Normal, mean 10, sd 1
y <- x + 0.1
z <- x + 0.2

kruskal.test(list(x, y, z))  #p = 3.954 e-05


I'm not sure that is what you want.

You might try robust regression or quantile regression.

• Hello! Thanks for sharing those new methods; they're intriguing. However, they might be too complex for my current goal. Please correct me if I'm wrong. I'm comparing group scores to explore the group's impact as a confounding factor. I've considered @Yacine Hajji's observations on large group sizes. Assuming the central limit theorem applies, I could treat data as normally distributed. Using aov() for one-way ANOVA, I'm still perplexed by the p-value of 7.61e-07. I'm unsure if the issue is my code or statistical approach. Any further insights are appreciated. Thanks for your help! Commented Jul 21, 2023 at 18:34
• You can't use the central limit theorem to get around the assumptions of ANOVA (or OLS regression). Sorry. OLS regression assumes normal errors. However, that assumption is only needed for some aspects of the results. And the question is whether the KW test does what you want. If it doesn't then you shouldn't use it. And if the right methods are too complex, then, I think you should hire a consultant (or, if you are in academia, maybe get someone in the stats department to be co-author). Commented Jul 21, 2023 at 18:55