0
$\begingroup$

I have two distributions derived from experiment - one has many more observations than the other, but their normalized distributions look similar. The data is binned, and I want to check if these distributions are similar or significantly different. Let's pretend the bin counts for both look like this;

Data_1 = [29573 28818 26900 25056 23794 22768 22067 21052 19830 19212]

Data_2 = [7680 7826 7170 6661 6373 6124 5790 5557 5327 5160]

These are not normal distributions, so I presume I need a non-parametric test. If I do a Two-sample Kolmogorov-Smirnov test in MATLAB on these two vectors, I get the null hypothesis rejected with $p = 1.88 \times 10^{-5}$. However, if I normalize these so that

Data_1_norm = [0.1237 0.1205 0.1125 0.1048 0.0995 0.0952 0.0923 0.0881 0.0829 0.0804]

Data_2_norm = [0.1206 0.1229 0.1126 0.1046 0.1001 0.0962 0.0909 0.0873 0.0837 0.0810]

and then do a two sample KS-test, I get null hypothesis not rejected ($p \approx 1 $). Can anyone help me understand which is the correct approach, and why the two disagree so much? Is the KS test the best metric for this? For what it's worth it is my suspicion that the data from both is probably the same, but I would have thought KS normalizes for the number of observations?

$\endgroup$
1
$\begingroup$

First of I would suggest that any test you perform you don't use binned data but the raw data. If you still want to stick to the bins, you need to take into account that the KS null hypothesis is that the 2 distributions are not different not only in shape, but also in scale and variance. Therefore, with the not normalised data, being the means of the two populations so different, the KS test says that most likely they come from different populations (low p-value), because you are not only looking at the shape...

If you normalise the data, then scale doesn't affect the test anymore, and the KS test says that it cannot be said that the samples come from different distributions.

To ilustrate this, look at this example (code provided in R):

> ks.test(Data_1_norm, Data_2_norm)

Two-sample Kolmogorov-Smirnov test

data:  Data_1_norm and Data_2_norm
D = 0.1, p-value = 1
alternative hypothesis: two-sided

> ks.test(Data_1_norm,Data_2_norm * 2)

Two-sample Kolmogorov-Smirnov test

data:  Data_1_norm and Data_2_norm * 2
D = 1, p-value = 1.083e-05
alternative hypothesis: two-sided

> ks.test(Data_1_norm, Data_2_norm * 1000)

Two-sample Kolmogorov-Smirnov test

data:  Data_1_norm and Data_2_norm * 1000
D = 1, p-value = 1.083e-05
alternative hypothesis: two-sided

You can see that any other comparison between the two samples that are not normalised are always rejecting the null hypothesis with the same p-value.

Suggestions: Use raw data or always normalise if you are interested in the shape of the binned data.

$\endgroup$
  • $\begingroup$ Thanks for this - that makes sense; with the raw data (using the full array, not just the 10 entries I've shown here, I get the following statistics: Mean data 1: 707.34 Mean data 2: 721.22 Median data 1: 487.29 Median data 2: 492.28 Which are fairly close - and you're right, when I bin them, this utterly skews the statistics! However, I think that while I would prefer to work with the raw data, I may have to bin them; the modes are Date 2 mode: 0.0081 Data 2 mode: 0.10 However - the image analysis algorithm is only accurate to +/ 25 - so the raw data might mislead... $\endgroup$ – DRG Jul 24 '17 at 15:47
  • $\begingroup$ ...as it could potentially see a difference where there really is none, due to the limitations of the measurements. In this case, is normalization after binning appropriate, given the limits of what the algorithm can detect? Thanks again for all the advice... $\endgroup$ – DRG Jul 24 '17 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.