5
$\begingroup$

I am working with two sets of data and would like to determine if they come from the same distribution -- hence the two population KS-test. At this point, I have binned both sets of data, counting the number of entries in each bin, to create relative frequency histograms. These binned arrays look like:

dis1 = [0.0, 4.0, 9.0, 10.0, 23.0, 26.0, 43.0, 44.0, 38.0, 4.0]
dis2 = [116.0, 160.0, 209.0, 247.0, 415.0, 455.0, 382.0, 288.0, 161.0, 44.0].

When I perform the KS-test for 2 datasets, I get:

[h,p]=kstest2(dis1,dis2)
h = 1
p = 1.7012e-04

meaning that the null hypothesis was rejected and the samples do not come from the same distribution.

Now, if I normalize the binned data with respect to the maximum in each array, I have:

norm_dis1=[0.0, 0.090909090909090912, 0.20454545454545456, 0.22727272727272727, 0.52272727272727271, 0.59090909090909094, 0.97727272727272729, 1.0, 0.86363636363636365, 0.090909090909090912]
norm_dis2 = [0.25494505494505493, 0.35164835164835168, 0.45934065934065932, 0.54285714285714282, 0.91208791208791207, 1.0, 0.83956043956043958, 0.63296703296703294, 0.35384615384615387, 0.096703296703296707],

such that now the norm_dis arrays vary from [0,1]. The 2 population KS-test for these arrays gives:

[h,p]=kstest2(norm_dis1,norm_dis2)
h = 0
p = 0.3129,

such that the null hypothesis is accepted and the populations come from the same distribution.

These two results are seemingly discrepant. What is the statistical difference between normalizing and not normalizing? And which is the correct statistical analysis of these two populations?

Thanks for your help.

Improve this question
$\endgroup$
9
  • 4
    $\begingroup$ Why are you binning the data? This would seem to be unnecessary indeed unhelpful. I'm not sure what software you're using but are you sure kstest2 works the way you think it does? Perhaps it's expecting the actual data, not a set of bins. $\endgroup$
    – Peter Ellis
    Commented Apr 17, 2012 at 4:51
  • $\begingroup$ I'm binning the data because I want to determine if the population histograms are similar; this is rather common in my field. The data itself is not binned - dis1 and dis2 are the number of "things" within each bin. $\endgroup$
    – cosmosis
    Commented Apr 17, 2012 at 5:05
  • $\begingroup$ It seems like the kstest of the binned data is now just telling you that the two distributions have a similar shape- the means of your two original samples are still hugely discrepant, but the spread of scores around those two means is similar (in the sense that the ratio of mean1:mean2 is similar to variance1:variance2) $\endgroup$
    – Marius
    Commented Apr 17, 2012 at 5:31
  • $\begingroup$ @Marius - I'm afraid I don't understand. The kstest is telling me that the distributions are both similar and not similar, depending on whether I normalize or not. $\endgroup$
    – cosmosis
    Commented Apr 17, 2012 at 6:08
  • 6
    $\begingroup$ Cosmo, that's because your normalizing changes the locations and scales so that you have two new distributions guaranteed to be approximately alike by virtue of that normalization. But please pay attention to @Peter's comment: the KS test is designed for unbinned data and doesn't really work properly with binned data to begin with. (You don't even give it information about bin counts, which is crucial.) Do not be overly concerned with what's common in your field: it is not in the least unusual for some (apparently) common practices to be statistically inferior--or worse. $\endgroup$
    – whuber ♦
    Commented Apr 17, 2012 at 6:13

1 Answer 1

Reset to default
4
$\begingroup$

Consider 2 population distributions N(10,1) and N(20,1), the KS test will say that data from those 2 distributions are different (since they have very different means relative to their standard deviations), now subtract the mean from each set of data and you have 2 groups that are approximately N(0,1), which the KS test may not be able to tell apart.

The same things is probably happening with your data, you are just doing a different normalization. Remember that the null hypothesis for the KS test is that the 2 distributions are identical in all ways (not just the same shape) so a difference in location, scale, or shape means that the null hypothesis is false.

And as others have said, do the test on the raw data, not on the bins.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Does it mean one should take the, i.e., average of just one set, and normalize both sets by that average? $\endgroup$
    – Py-ser
    Commented Apr 11, 2014 at 9:38
  • $\begingroup$ @Py-ser, why normalize at all? it is not a requirement of the KS test. $\endgroup$
    – Greg Snow
    Commented Apr 14, 2014 at 20:36
  • 1
    $\begingroup$ normalization is standard in showing data/compare different sets, even if this is not required by the test. Anyway, I was asking that because is of my interest to know that, and because you mentioned in your topic "you are just doing a different normalization". So I was asking what was the correct way to normalize in this case. $\endgroup$
    – Py-ser
    Commented Apr 15, 2014 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.