2
$\begingroup$

I am trying to investigate the differences between two datasets. Each one is composed of 20 000 elements obtained from a Monte-Carlo method whose implementation slightly differs in both cases. I'd like to draw a conclusion regarding the influence of this change over my results.

As for now, I have performed two-sample Kolmogorov-Smirnov and Anderson-Darling tests, using scipy library. Comparing outcomes to critical values, it seems like both tests disagree; no significant difference is found for KS, while AD-distance is greater than the corresponding critical value.

Aware of the fact that AD test performs well for huge tailed distributions, I compared the $95^{th}$ percentile value of both sets, hoping for a significant difference, in vain.

Is there anything I should look for to explain this difference ?

Improve this question
$\endgroup$
2
  • $\begingroup$ (1) There are many situations in which the K-S test has poor power to detect that two samples come from slightly different populations. Yours may be one of these. (2) Goodness-of-fit tests on large samples may sometimes find differences that are real, but unimportant. Maybe you shouldn't be using GOF tests to determine whether your two Monte Carlo methods are different in ways that are of practical importance. (3) I don't understand your comment on 95th percentiles near the end of your Question. If that part is important to you, maybe give a more detailed explanation. $\endgroup$
    – BruceET
    Jul 26, 2020 at 22:37
  • $\begingroup$ Thanks ! About the 95th percentile, I supposed that if the AD test detected a difference while the KS test did not, it was probably because the distribution obtained with the second Monte-Carlo method was fat-tailed. I supposed that in that case, a high percentile statistics would differ between both distributions. Maybe this is not a good idea to explain how the distributions differ.. $\endgroup$
    – Clej
    Jul 26, 2020 at 23:41

1 Answer 1

Reset to default
2
$\begingroup$

I realize that this may not be anything like what you are doing, but the following simulation in R shows some quirks of goodness-of-fit tests.

I generate $n = 5000$ observations z1 from Student's t distribution with 10 degrees of freedom, which is similar in shape to standard normal but has fatter tails (and variance 1.25 instead of 1). Also, 5000 observations z2 from standard normal.

  • A K-S two-sample test does not detect a difference between z1 and z2,
  • A one-sample K-S test detects that z1 is not from standard normal.
  • A Shapiro-Wilk test detects that z1 is not from any normal distribution.

,

set.seed(726)
z1 = rt(5000,10);  z2 = rnorm(5000)
ks.test(z1,z2)$p.val
[1] 0.1938702
ks.test(z1,pnorm)$p.val
[1] 0.006897905
shapiro.test(z1)$p.val
[1] 4.39737e-12

Relevant graphs:

enter image description here

par(mfrow=c(1,2))
 hist(z1, prob=T, br=30, ylim=c(0,.4), col="skyblue2", 
      main="T(10) Sample, Standard Normal Density")
  curve(dnorm(x), add=T, col="red")
 qqnorm(z1, main="Q-Q Plot of T(10) Sample")
  qqline(z1, col="green3", lwd=2)
par(mfrow=c(1,1))
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for that remark, it is quite impressive. Could it be that the distributions still look alike ? I doubt it though, 5000 elements are a lot, isn'it ? $\endgroup$
    – Clej
    Jul 26, 2020 at 23:44
  • $\begingroup$ Tried comparing ECDF plots of z1 and z2 on same axes. Essentially, impossible to distinguish them. K-S test statisitic $D$ is max vertical discrepancy between the two ECDFs. // Maybe just 'unlucky' samples of size 5000. More extensive simulation could approximate power of K-S test to distinguish. $\endgroup$
    – BruceET
    Jul 26, 2020 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.