# kde.test gives different p-values for similar KDEs

I have 4 sets of random floats between [0,1]: $d1, d2, d3, d4$. I need to compare these sets by pairs and I make use of the kde.test function of R's ks package to do so.

When I use kde.test to obtain the pvalue of distributions $d1,d2$ I get a value of ~0.55, but if I do the same for $d3,d4$ I get ~0.00.

Here's the code I use in R to come up with those values:

d1 <- c(...)
d2 <- c(...)
pv12 <- kde.test(x1=d1, x2=d2)$pvalue d3 <- c(...) d4 <- c(...) pv34 <- kde.test(x1=d3, x4=d2)$pvalue


The complete $d1,d2,d3,d4$ sets are here and their lengths are: len(d1) = 20, len(d2) = 210, len(d3) = 200, len(d4) = 2100.

Sets $d1,d2$ have fewer elements than $d3,d4$ but the shapes of the KDEs are similar. Here's the KDEs for $d1,d2$ (blue and red respectively): and here's the KDEs for $d3,d4$ (blue and red respectively): Because of the similarity of the KDEs I would have expected somewhat similar pvalues but the results I get are wildly different.

I'm either not understanding what the kde.test does or what the resulting pvalues means (or both).

Could someone explain what is happening and why my expectations are no correct?

I get very similar results if I apply the Kolmogorov-Smirnov test from the scipy.stats.ks_2samp package which also gives a p-value as output.

• What are the relative sample sizes? Dec 2 '13 at 17:06
• @Glen_b I'm sorry, I don't know what that is. I have zero training in statistics. If you explain me I'll gladly look it up. Dec 2 '13 at 17:09
• How many values are used to find the four d curves? How big is each 'set'? Dec 2 '13 at 17:16
• This is a widespread phenomenon whereby significance tests from larger sample sizes give lower P-values for otherwise equivalent results. Consider as a simpler example: which of these is a stronger refutation of pr(heads) = 0.5: 7/10 heads, 70/100, 700/1000? Dec 2 '13 at 19:27
• Exactly as Nick suggests (and that was why I asked) - the lower p-value in the second case (even though the differences in density estimate look similar) will be largely due to the larger sample size, since a more precise estimate serves to reduce the possibility that the observed difference is due to random variation. Dec 2 '13 at 22:13