9
$\begingroup$

I have two samples where I want to tell if their distributions are significantly different. I conducted a Kolmogorov-Smirnov test which rejected the null hypothesis (D=0.0983, p=2.14e-11) leading me to believe that the samples do indeed come from different distributions. The problem is, how can I determine what is significantly different about these samples? Are the distributions different in mean? skew? something else? Examining the histograms with each other do not give any immediate indication of difference. Sure, the red sample has a bit lower values than the blue sample, but is that what caused the KS test to be significant? How can I know what caused the KS test null hypothesis to fail?

Sample size of red = 1306 Sample size of blue = 646,513

Red vs blue samples

ECDFs compared

Improve this question
$\endgroup$
1
8
$\begingroup$

The K-S is an omnibus test. It doesn't identify the form of difference.

You can identify the place or places where the difference in ECDF is largest but that doesn't always tell you much.

You can spot some kinds of change in the ECDF. Here's some examples:

enter image description here

If you look at the ECDFs in your post, you can see the blue cdf is lower than the red across pretty much the entire range (which implies the blue values are typically larger*), though the shape is broadly similar.

* You can also see that in the histogram where the pink is higher in the leftmost bar, but past 300 the blue bars are typically a little taller.

So anyway, that's how I'd describe the main difference in distributions (somewhat similar in general shape, but the values in the blue category tend to be a bit larger on average). I wouldn't characterize it as either a location shift or a scale-shift.

One thing that worries me is that the distribution looks quite discrete. If so, the p-value from the Kolmogorov-Smirnov test isn't particularly meaningful.

What's your sample size?

Improve this answer
$\endgroup$
6
  • $\begingroup$ No, the distribution is not discrete, it is just poorly sampled at higher values. As added above, sample size is 1306. $\endgroup$ – CephBirk Dec 9 '14 at 2:43
  • 1
    $\begingroup$ If the distribution is not discrete where are there so few jumps in the ECDF? (I count only 18 in the blue series) There should be hundreds of them, one per data value. $\endgroup$ – Glen_b Dec 9 '14 at 3:07
  • 1
    $\begingroup$ If the data are actually continuous, the sizes of the jumps should also be tiny, not large. The plot clearly shows a discrete distribution. If the data are not discrete, what are we looking at instead of your data? Also, you should have two sample sizes, one for the pink one for the blue... unless they're paired, in which case, you shouldn't be using a KS test to compare them. $\endgroup$ – Glen_b Dec 9 '14 at 3:18
  • 1
    $\begingroup$ No, I'm sorry, I've been doing a few of these tests at once and got them confused! My apologies. You are correct, the data are discrete which is why I am using the dgof::ks.test() function which corrects the p value for discrete data, to my understanding. $\endgroup$ – CephBirk Dec 9 '14 at 3:23
  • $\begingroup$ Why exactly the same? Are they paired? $\endgroup$ – Glen_b Dec 9 '14 at 3:50
1
$\begingroup$

As far as I know, the (two sample) K-S test is not that helpful in telling you how the two distributions differ, as it compares both location and shape. If you think the difference lies in location, run a Mann-Whitney. If you think it lies in shape...that's more complicated. Some people calculate GINI to compare skewness. Others recommend running bootstraps, so you get (say) 10,000 skewness and kurtosis scores per sample, and then you can just run a t-test to compare them. This has always seemed a bit odd to me, though.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.