3
$\begingroup$

I have some data (called egecNonmated) in R and I am trying to show that the distributions of a variable (MatchScore) are identical across three different Categories (Cat). I am using a box plot to do this visually but it has been suggested (wisely) that I also show this quantitatively. Below is the R code I am running to do the visual analysis:

allNm <- boxplot(MatchScore~Cat, data=egecNonmated)

Showing "allNm" produces the following graph:

RBoxplot

Viewing the statistics associated with the boxplot via

allNm$stats

Produces the following:

          [,1]      [,2]      [,3]
[1,] 0.3988655 0.3982853 0.3988191
[2,] 0.4349543 0.4347747 0.4349721
[3,] 0.4478581 0.4478713 0.4479012
[4,] 0.4590152 0.4591016 0.4590747
[5,] 0.4951032 0.4953164 0.4952006

We can see that the median score [3,] for each of the categories is identical to three decimal places. As are all the other statistics (lower whisker [1,], lower hinge [2,], upper hinge [4,] and upper whisker [5,]. So I would expect kruskal wallis to accept the null hypotheis that the three distributions are from identical populations. However when I run

kruskal.test(MatchScore ~ Cat, data=egecNonmated)

I get the following:

Kruskal-Wallis chi-squared = 15.5941, df = 2, p-value = 0.0004109

If I am interpreting this correctly tells me that at a p value of 0.01 I should reject the null hypothesis and therefore these distributions are NOT identical. Am I using the wrong test? Wrong interpretation? Thanks in advance for the help

Improve this question
$\endgroup$
5
  • $\begingroup$ The third row is not the mean, but the median. $\endgroup$
    – Glen_b
    Commented Nov 9, 2014 at 23:10
  • $\begingroup$ i might take a different tack than the answers below. Boxplots can hide significant differences in a variable when n is large. it is often worth - using domain knowledge - trying to think of different ways of presenting the data (tables or subgroup graphs) $\endgroup$
    – charles
    Commented Nov 10, 2014 at 7:16
  • $\begingroup$ @charles: Plotting the empirical distribution functions & kernel-smoothed density estimates can be useful. $\endgroup$
    – Scortchi ♦
    Commented Nov 10, 2014 at 9:49
  • $\begingroup$ You might also consider looking at measures of association $\endgroup$
    – bdeonovic
    Commented Nov 11, 2014 at 1:40
  • $\begingroup$ @Scortchi - Thank you for the recommendation. I've just started using kernal-smoothers and violin-plots. Still have a lot to learn before I have a good handle on them. $\endgroup$
    – charles
    Commented Nov 11, 2014 at 12:33

1 Answer 1

Reset to default
6
$\begingroup$

I don't think anything is amiss here.

The three distributions are very similar, but not identical (why would they be exactly the same, rather than different in very tiny ways?)

Your sample sizes are very large, so your hypothesis test can pick up extremely small differences, even trivial ones.

It's giving you information, but it's probably an answer to a question you don't actually care about very much (which suggests that you probably didn't actually want to test for identical distributions in the first place; it's likely you need a different tool for whatever your actual problem is).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Hi Glen. So I think "identical" was maybe a bit strong. I just want to quantify the similarity in the distributions. I thought kruskal wallis was the tool to do that but clearly not. Do you know of a statistical method that is similar but will work on sets with large N? $\endgroup$
    – JHowIX
    Commented Nov 9, 2014 at 23:42
  • 2
    $\begingroup$ Kruskal Wallis works perfectly on large $n$. Your problem isn't that the screwdriver doesn't work - it's great unless you try to hammer in nails with it. You need a different tool than null-hypothesis significance tests, but first you need to clearly identify the problem you want to solve. It might be best achieved by finding confidence intervals for location-shifts. It might be best achieved by some overall measure of effect size. It might be handled by equivalence tests. It might need something else ... which you might want depends on just what you're trying to achieve in the end. $\endgroup$
    – Glen_b
    Commented Nov 10, 2014 at 0:08
  • 2
    $\begingroup$ @JHowIX I think it would be worthwhile for you to visit the question Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples? $\endgroup$
    – Alexis
    Commented Nov 10, 2014 at 6:20
  • $\begingroup$ @Alexis - Yes, this is exactly the explanation I needed. Both yours an Glen_b's explanations have had me leaning toward using TOST (although my data isn't strictly normally distributed, but its close). I am going to try that out this week instead of KW. Thanks for the help. $\endgroup$
    – JHowIX
    Commented Nov 10, 2014 at 14:40
  • $\begingroup$ @JHowIX You can apply TOST for nonparametric tests using z test approximations, it's a pretty straightforward translation of the approach from t to z. $\endgroup$
    – Alexis
    Commented Nov 10, 2014 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.