Two-sample Kolmogorov-Smirnov, Anderson-Darling, and after? I am trying to investigate the differences between two datasets. Each one is composed of 20 000 elements obtained from a Monte-Carlo method whose implementation slightly differs in both cases. I'd like to draw a conclusion regarding the influence of this change over my results.
As for now, I have performed two-sample Kolmogorov-Smirnov and Anderson-Darling tests, using scipy library. Comparing outcomes to critical values, it seems like both tests disagree; no significant difference is found for KS, while AD-distance is greater than the corresponding critical value.
Aware of the fact that AD test performs well for huge tailed distributions, I compared the $95^{th}$ percentile value of both sets, hoping for a significant difference, in vain.
Is there anything I should look for to explain this difference ?
 A: I realize that this may not be anything like what you
are doing, but the following simulation in R shows some
quirks of goodness-of-fit tests.
I generate $n = 5000$ observations z1 from Student's t distribution with 10 degrees
of freedom, which is similar in shape to standard normal but has fatter tails (and variance 1.25 instead of 1). Also, 5000 observations z2 from standard normal.

*

*A K-S two-sample test does not detect a difference between z1 and z2,

*A one-sample K-S test detects that z1 is not from standard normal.

*A Shapiro-Wilk test detects that z1 is not from any normal distribution.

,
set.seed(726)
z1 = rt(5000,10);  z2 = rnorm(5000)
ks.test(z1,z2)$p.val
[1] 0.1938702
ks.test(z1,pnorm)$p.val
[1] 0.006897905
shapiro.test(z1)$p.val
[1] 4.39737e-12

Relevant graphs:

par(mfrow=c(1,2))
 hist(z1, prob=T, br=30, ylim=c(0,.4), col="skyblue2", 
      main="T(10) Sample, Standard Normal Density")
  curve(dnorm(x), add=T, col="red")
 qqnorm(z1, main="Q-Q Plot of T(10) Sample")
  qqline(z1, col="green3", lwd=2)
par(mfrow=c(1,1))

