1
$\begingroup$

I have two sets of random floats between [0,1], each one with a different number of elements:

$$A = \{0.3637852, 0.2330702, 0.1683102, 0.2127219, 0.0152532, ..., N_A\}$$ $$B = \{0.4541056, 0.7521812, 0.0266602, 0.5099002, 0.3468181, ..., N_B\}$$

where $N_B > N_A$.

I need to asses the similarity between these two sets disregarding the difference in number of elements (ie: the fact that $N_B > N_A$ should not play a role in the similarity assesment) since I'm only interested in how the values are scattered between [0,1], not how many of them there are in each set.

So far I've applied the 1D kde.test available in Duong's R package 'ks' (see page 27) which returns a p-value, and I've also applied the python function scipy.stats.ks_2samp which is a 1D Kolmogorov-Smirnov statistic which returns a "KS statistic" and a "two-tailed p-value".

My questions are:

1- Is one of these statistics (KDE's p-value, KS statistic or the two-tailed p-value) recommended for my needs? If so, why?

2- What is the difference between the "KS statistic" and a "two-tailed p-value"?

3- Will the difference in the number of elements in each set affect the outcome of these statistics? If so, how can I avoid that? Is it even possible?

$\endgroup$
4
  • 3
    $\begingroup$ Have you considered a Q-Q plot? It will truly help you assess the similarity rather than merely testing the hypothesis that they are drawn from the same distribution. $\endgroup$
    – whuber
    Nov 27 '13 at 20:23
  • $\begingroup$ First time I've heard of a Q-Q plot. Does it give you a number quantifying the assessment? Because I really need one. $\endgroup$
    – Gabriel
    Nov 27 '13 at 20:25
  • $\begingroup$ It does far better than a single number: it produces a graph showing how the datasets differ in detail. From this graph a whole host of numbers can be produced, depending on how you wish to measure the dissimilarity of the datasets. $\endgroup$
    – whuber
    Nov 27 '13 at 20:29
  • $\begingroup$ @whuber please see stats.stackexchange.com/questions/77907/…, I'm a bit lost. Thank you. $\endgroup$
    – Gabriel
    Nov 27 '13 at 22:34
3
$\begingroup$

1- Is one of these statistics (KDE's p-value, KS statistic or the two-tailed p-value) recommended for my needs? If so, why?

Your needs as expressed do not seem to be sufficiently clearly defined as to differentiate between them. They both test for a difference in distribution.

2- What is the difference between the "KS statistic" and a "two-tailed p-value"?

The two sample Kolmogorov-Smirnov statistic is the largest difference in ECDFs for the two samples:

enter image description here

(The data here is the same data I generated for your other question. Here the A sample is red and the B sample is blue.)

The height difference in ECDFs at x=35 is 1/6 or about 0.1667 (indeed anywhere in $[34.50717,35.32252)$ ), the same as the value produced by calculating the statistic:

> ks.test(A,B)

        Two-sample Kolmogorov-Smirnov test

data:  A and B 
D = 0.1667, p-value = 0.6228
alternative hypothesis: two-sided 

The meaning of the p-value is as for any hypothesis test - the probability of obtaining a statistic at least as unusual (in this case, at least as large) if the null hypothesis were true.

3- Will the difference in the number of elements in each set affect the outcome of these statistics?

No, the KS test, and (to my understanding) the KDE-based one both handle different sample sizes.

Here's approximate data values, in case anyone needs them

> print(A,d=3)
 [1] 41.34 25.92 55.30 50.06 75.67  3.03 61.81 34.51 34.33  9.62 94.95 24.73
[13] 30.41 11.77 25.13 90.75 12.62 36.14 56.91 29.76 15.34 62.58 33.03 36.44
[25] 47.90 66.01 42.49 18.21 31.58 58.30 17.63 70.81 73.86 46.63 10.24 12.02
[37] 47.14 15.56 80.27 12.76 33.61 52.08 41.64 13.19 32.96 64.21 81.15 32.37
[49] 33.79 40.43
> print(B,d=3)
 [1] 39.43 57.93 72.91 12.81  3.76 39.02 56.02 40.28 30.25 75.31  2.46 81.44
[13] 11.74  9.32 60.85 75.39 44.58 62.05 53.33 63.63 29.90 31.41 59.82 50.37
[25] 41.17 49.49 20.34 35.32 33.82 35.47
>

All that said, I recommend you consider whuber's words most carefully. There's a lot of good advice packed into very few words.

$\endgroup$
1
  • $\begingroup$ Thank you very much @Glen_b, I'll certainly consider whuber's comment. $\endgroup$
    – Gabriel
    Nov 29 '13 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.