kde.test gives different p-values for similar KDEs

Question

I have 4 sets of random floats between [0,1]: $d1, d2, d3, d4$. I need to compare these sets by pairs and I make use of the kde.test function of R's ks package to do so.

When I use kde.test to obtain the pvalue of distributions $d1,d2$ I get a value of ~0.55, but if I do the same for $d3,d4$ I get ~0.00.

Here's the code I use in R to come up with those values:

d1 <- c(...)
d2 <- c(...)
pv12 <- kde.test(x1=d1, x2=d2)$pvalue

d3 <- c(...)
d4 <- c(...)
pv34 <- kde.test(x1=d3, x4=d2)$pvalue

The complete $d1,d2,d3,d4$ sets are here and their lengths are: len(d1) = 20, len(d2) = 210, len(d3) = 200, len(d4) = 2100.

Sets $d1,d2$ have fewer elements than $d3,d4$ but the shapes of the KDEs are similar. Here's the KDEs for $d1,d2$ (blue and red respectively):

and here's the KDEs for $d3,d4$ (blue and red respectively):

Because of the similarity of the KDEs I would have expected somewhat similar pvalues but the results I get are wildly different.

I'm either not understanding what the kde.test does or what the resulting pvalues means (or both).

Could someone explain what is happening and why my expectations are no correct?

---

Add

I get very similar results if I apply the Kolmogorov-Smirnov test from the scipy.stats.ks_2samp package which also gives a p-value as output.

Answer 1

This is another example of a widespread phenomenon, whereby significance tests from larger sample sizes give lower P-values for otherwise equivalent results.

Consider a simpler example from tossing a coin, with two outcomes, heads or tails. Which of these is a stronger refutation of pr(heads) = 0.5: 7/10 heads, 70/100, 700/1000? Intuitively it should appear that 7/10 (or more) is much more likely to be seen as a chance fluctuation than 700/1000 (or more) if the probability of heads is really 0.5. The observed fraction is the same, but it is only part of the information.

The sample sizes are key here because they indicate how much data went into the estimates. You are not comparing PDFs. No one knows the correct PDFs for empirical variables; you have to estimate them.

So the logic here is just standard logic for significance tests. You are using such a test, and are constrained by its logic.

A different issue is that kernel density estimation at least by default typically smooths some mass beyond finite ranges, here [0,1]. In your case this is more evident at 0. Programs often ignore this lost mass. I have no idea what the procedure you used does about this.