1
$\begingroup$

I have two numeric data vectors: NA_Sales and EU_Sales, which are really similar. When I plot their ECDFs like this:

plot(ecdf(dataset$NA_Sales))
lines(ecdf(dataset$EU_Sales), col="red")

Then I get this diagram:

enter image description here

Which shows they are really similar. But when I run the Kolmogorov-Smirnov test like this:

ks.test(dataset$NA_Sales,dataset$EU_Sales)

or

ks.test(dataset$NA_Sales,dataset$EU_Sales)$p

Then I get a p-value equal to zero. Why? The diagram shows they are similar distributions. Isn't ks.test() supposed to tell this by their CDF? I should get a p-value higher than 0.05.

$\endgroup$

migrated from stackoverflow.com May 6 at 22:56

This question came from our site for professional and enthusiast programmers.

  • 3
    $\begingroup$ With any consistent hypothesis test, very small p-values occur with small effect sizes when sample sizes are sufficiently large. They don't answer the question "are these similar?". In particular, with the KS test any small difference will be significant at a large enough sample size. Start with the discussion under this question: stats.stackexchange.com/questions/349618/… and discussion on the same issue in the one sample KS test here: stats.stackexchange.com/questions/74434/… $\endgroup$ – Glen_b May 7 at 0:10
  • 1
    $\begingroup$ For an explanation of this issue with hypothesis tests in general see stats.stackexchange.com/questions/2516/… and stats.stackexchange.com/questions/125750/sample-size-too-large; note, for example that the 5% critical value for the two-sample test is $1.36\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}$ (&the critical value grows very slowly in $1/α$); as $\min(n_1,n_2)$ increases, that critical value gets smaller and smaller, so for a specific distance between your samples, the same test statistic will give smaller and smaller p-values $\endgroup$ – Glen_b May 7 at 0:20
  • 1
    $\begingroup$ You may also get some value from this answer: stats.stackexchange.com/questions/136999/… since the first two paragraphs apply equally here, and from some of the answers here: stats.stackexchange.com/questions/82579/… (even though that relates to the one sample test, the issues are again the same) $\endgroup$ – Glen_b May 7 at 0:34
  • 3
    $\begingroup$ These plots are greatly different, but your mode of displaying them hides the differences. Draw a P-P plot instead. $\endgroup$ – whuber May 7 at 1:05
  • 2
    $\begingroup$ whuber is quite right -- the differences are considerably more substantial than they look in this plot -- the upper quantiles in first variable are typically about 50% larger than in the second, the medians are even more different, while the lower quartiles are both 0). But even if they were much more alike, at n1=n2=16598 even quite small differences in cdf would be significant. You should note two things that make the KS-test p-value calculation problematic - these data are paired, and they have a lot of zeros and quite a lot of tied small values $\endgroup$ – Glen_b May 7 at 1:37
2
$\begingroup$

Let us show both the issue that whuber raised (that the difference is large, not small) and the one I raised (that very small differences will be significant) in one figure -- by the simple expedient of using a monotonic transformation of the data to tame the long right tail.

In this case, the transformation $g(x) = x/(1+x)$ will pull all of the data into $[0,1)$, while leaving the Kolmogorov-Smirnov distance unchanged - as any transformation that is strictly monotonic in the range of the data would do.

Plot of transformed ecdf for both data sets with maximum KS distance (0.21358) marked in

With 16598 data points, this size of difference is far into the extreme upper tail. The tiny 5% critical distance marked on the plot is that under the assumptions of the test (so it's for continuous data).

The size of the critical distance grows very slowly in $1/\alpha$; if I have calculated it correctly, just doubling the size of that small critical difference would correspond to a significance level of less that $10^{-5}$ for continuous data.

However, because these data are discrete, that's highly conservative (i.e. actually corresponds to a much lower significance level); because of that any calculated p-values are considerably too large.

Note also that these data are paired; that also violates the independence assumption of the test (indeed a plot shows they're substantially correlated). Because of that and the discreteness issue, you really can't put any weight on the actual numeric p-value given by the test -- it's not a meaningful number here.

Further, these are data over time in a changing market; the values within each variable represent a mixture of different distributions, and they are also not going to be independent of each other. The value in trying to use a formal test in this way is highly dubious.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.