Wrong Intepretation of Kruskal Wallis Test in R

I have some data (called egecNonmated) in R and I am trying to show that the distributions of a variable (MatchScore) are identical across three different Categories (Cat). I am using a box plot to do this visually but it has been suggested (wisely) that I also show this quantitatively. Below is the R code I am running to do the visual analysis:

allNm <- boxplot(MatchScore~Cat, data=egecNonmated)


Showing "allNm" produces the following graph:

Viewing the statistics associated with the boxplot via

allNm$stats  Produces the following:  [,1] [,2] [,3] [1,] 0.3988655 0.3982853 0.3988191 [2,] 0.4349543 0.4347747 0.4349721 [3,] 0.4478581 0.4478713 0.4479012 [4,] 0.4590152 0.4591016 0.4590747 [5,] 0.4951032 0.4953164 0.4952006  We can see that the median score [3,] for each of the categories is identical to three decimal places. As are all the other statistics (lower whisker [1,], lower hinge [2,], upper hinge [4,] and upper whisker [5,]. So I would expect kruskal wallis to accept the null hypotheis that the three distributions are from identical populations. However when I run kruskal.test(MatchScore ~ Cat, data=egecNonmated)  I get the following: Kruskal-Wallis chi-squared = 15.5941, df = 2, p-value = 0.0004109  If I am interpreting this correctly tells me that at a p value of 0.01 I should reject the null hypothesis and therefore these distributions are NOT identical. Am I using the wrong test? Wrong interpretation? Thanks in advance for the help • The third row is not the mean, but the median. – Glen_b Nov 9 '14 at 23:10 • i might take a different tack than the answers below. Boxplots can hide significant differences in a variable when n is large. it is often worth - using domain knowledge - trying to think of different ways of presenting the data (tables or subgroup graphs) – charles Nov 10 '14 at 7:16 • @charles: Plotting the empirical distribution functions & kernel-smoothed density estimates can be useful. – Scortchi - Reinstate Monica Nov 10 '14 at 9:49 • You might also consider looking at measures of association – bdeonovic Nov 11 '14 at 1:40 • @Scortchi - Thank you for the recommendation. I've just started using kernal-smoothers and violin-plots. Still have a lot to learn before I have a good handle on them. – charles Nov 11 '14 at 12:33 1 Answer I don't think anything is amiss here. The three distributions are very similar, but not identical (why would they be exactly the same, rather than different in very tiny ways?) Your sample sizes are very large, so your hypothesis test can pick up extremely small differences, even trivial ones. It's giving you information, but it's probably an answer to a question you don't actually care about very much (which suggests that you probably didn't actually want to test for identical distributions in the first place; it's likely you need a different tool for whatever your actual problem is). • Hi Glen. So I think "identical" was maybe a bit strong. I just want to quantify the similarity in the distributions. I thought kruskal wallis was the tool to do that but clearly not. Do you know of a statistical method that is similar but will work on sets with large N? – JHowIX Nov 9 '14 at 23:42 • Kruskal Wallis works perfectly on large$n\$. Your problem isn't that the screwdriver doesn't work - it's great unless you try to hammer in nails with it. You need a different tool than null-hypothesis significance tests, but first you need to clearly identify the problem you want to solve. It might be best achieved by finding confidence intervals for location-shifts. It might be best achieved by some overall measure of effect size. It might be handled by equivalence tests. It might need something else ... which you might want depends on just what you're trying to achieve in the end. – Glen_b Nov 10 '14 at 0:08
• @JHowIX I think it would be worthwhile for you to visit the question Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples? – Alexis Nov 10 '14 at 6:20
• @Alexis - Yes, this is exactly the explanation I needed. Both yours an Glen_b's explanations have had me leaning toward using TOST (although my data isn't strictly normally distributed, but its close). I am going to try that out this week instead of KW. Thanks for the help. – JHowIX Nov 10 '14 at 14:40
• @JHowIX You can apply TOST for nonparametric tests using z test approximations, it's a pretty straightforward translation of the approach from t to z. – Alexis Nov 10 '14 at 19:14