0
$\begingroup$

I have 4 sets of random floats between [0,1]: $d1, d2, d3, d4$. I need to compare these sets by pairs and I make use of the kde.test function of R's ks package to do so.

When I use kde.test to obtain the pvalue of distributions $d1,d2$ I get a value of ~0.55, but if I do the same for $d3,d4$ I get ~0.00.

Here's the code I use in R to come up with those values:

d1 <- c(...)
d2 <- c(...)
pv12 <- kde.test(x1=d1, x2=d2)$pvalue

d3 <- c(...)
d4 <- c(...)
pv34 <- kde.test(x1=d3, x4=d2)$pvalue

The complete $d1,d2,d3,d4$ sets are here and their lengths are: len(d1) = 20, len(d2) = 210, len(d3) = 200, len(d4) = 2100.

Sets $d1,d2$ have fewer elements than $d3,d4$ but the shapes of the KDEs are similar. Here's the KDEs for $d1,d2$ (blue and red respectively):

d1 red, d2 blue

and here's the KDEs for $d3,d4$ (blue and red respectively):

d3 red, d4 blue

Because of the similarity of the KDEs I would have expected somewhat similar pvalues but the results I get are wildly different.

I'm either not understanding what the kde.test does or what the resulting pvalues means (or both).

Could someone explain what is happening and why my expectations are no correct?


Add

I get very similar results if I apply the Kolmogorov-Smirnov test from the scipy.stats.ks_2samp package which also gives a p-value as output.

| cite | improve this question | | | | |
$\endgroup$
  • $\begingroup$ What are the relative sample sizes? $\endgroup$ – Glen_b -Reinstate Monica Dec 2 '13 at 17:06
  • $\begingroup$ @Glen_b I'm sorry, I don't know what that is. I have zero training in statistics. If you explain me I'll gladly look it up. $\endgroup$ – Gabriel Dec 2 '13 at 17:09
  • $\begingroup$ How many values are used to find the four d curves? How big is each 'set'? $\endgroup$ – Glen_b -Reinstate Monica Dec 2 '13 at 17:16
  • 1
    $\begingroup$ This is a widespread phenomenon whereby significance tests from larger sample sizes give lower P-values for otherwise equivalent results. Consider as a simpler example: which of these is a stronger refutation of pr(heads) = 0.5: 7/10 heads, 70/100, 700/1000? $\endgroup$ – Nick Cox Dec 2 '13 at 19:27
  • 1
    $\begingroup$ Exactly as Nick suggests (and that was why I asked) - the lower p-value in the second case (even though the differences in density estimate look similar) will be largely due to the larger sample size, since a more precise estimate serves to reduce the possibility that the observed difference is due to random variation. $\endgroup$ – Glen_b -Reinstate Monica Dec 2 '13 at 22:13
3
$\begingroup$

This is another example of a widespread phenomenon, whereby significance tests from larger sample sizes give lower P-values for otherwise equivalent results.

Consider a simpler example from tossing a coin, with two outcomes, heads or tails. Which of these is a stronger refutation of pr(heads) = 0.5: 7/10 heads, 70/100, 700/1000? Intuitively it should appear that 7/10 (or more) is much more likely to be seen as a chance fluctuation than 700/1000 (or more) if the probability of heads is really 0.5. The observed fraction is the same, but it is only part of the information.

The sample sizes are key here because they indicate how much data went into the estimates. You are not comparing PDFs. No one knows the correct PDFs for empirical variables; you have to estimate them.

So the logic here is just standard logic for significance tests. You are using such a test, and are constrained by its logic.

A different issue is that kernel density estimation at least by default typically smooths some mass beyond finite ranges, here [0,1]. In your case this is more evident at 0. Programs often ignore this lost mass. I have no idea what the procedure you used does about this.

| cite | improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.