Kolmogorov-Smirnov scipy_stats.ks_2samp Distribution Comparison Newbie Kolmogorov-Smirnov question. I have 2 sample data set. When I apply the ks_2samp from scipy to calculate the p-value, its really small = Ks_2sampResult(statistic=0.226, pvalue=8.66144540069212e-23)
When I compare their histograms, they look like they are coming from the same distribution. Am I interpreting the test incorrectly?
On the scipy docs If the KS statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same. So with the p-value being so low, we can reject the null hypothesis that the distribution are the same right?
On a side note, are there other measures of distribution that shows if they are similar? Histogram overlap? KDE overlaps?
thanks,

 A: It looks like you have a reasonably large amount of data (assuming the y-axis are counts). The KS test (as will all statistical tests) will find differences from the null hypothesis no matter how small as being "statistically significant" given a sufficiently large amount of data (recall that most of statistics was developed during a time when data was scare, so a lot of tests seem silly when you are dealing with massive amounts of data). Assuming that your two sample groups have roughly the same number of observations, it does appear that they are indeed different just by looking at the histograms alone. For instance it looks like the orange distribution has more observations between 0.3 and 0.4 than the green distribution. That isn't to say that they don't look similar, they do have roughly the same shape but shifted and squeezed perhaps (its hard to tell with the overlay, and it could be me just looking for a pattern).
I should also note that the KS test tell us whether the two groups are statistically different with respect to their cumulative distribution functions (CDF), but this may be inappropriate for your given problem. For example, perhaps you only care about whether the median outcome for the two groups are different. You can have two different distributions that are equal with respect to some measure of the distribution (e.g. the median). Further, just because two quantities are "statistically" different, it does not mean that they are "meaningfully" different. That can only be judged based upon the context of your problem e.g., a difference of a penny doesn't matter when working with billions of dollars.
A: I can't retrieve your data from your histograms. So let's look at largish datasets
from a couple of slightly different distributions and see if the K-S two-sample test
can discern that the two samples aren't from the same distribution. [I'm using R.]
set.seed(801)
x1 = rbeta(1000, 11, 9)
x2 = rbeta(1000, 12, 8)

summary(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2311  0.4852  0.5532  0.5517  0.6234  0.8387 
summary(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2516  0.5278  0.6043  0.5994  0.6736  0.8953 

Here are histograms of the two sample, each with the density function of
its population shown for reference.

Somewhat similar, but not exactly the same. For example, $\mu_1 = 11/20 = 5.5$ and $\mu_2 = 12/20 = 6.0.$ Furthermore, the K-S test rejects the null hypothesis
that the two samples came from the same distribution. K-S tests aren't exactly
famous for their good power, but with $n=1000$ observations from each sample,
the test was able to reject with P-value very near $0.$
ks.test(x1, x2)

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x2
D = 0.205, p-value < 2.2e-16
alternative hypothesis: two-sided

The test statistic $D$ of the K-S test is the maximum vertical distance between the
empirical CDFs (ECDFs) of the samples.
plot(ecdf(x1), xlim=0:1, col="blue")
 lines(ecdf(x2), col="brown")


As seen in the ECDF plots, x2 (brown) stochastically dominates
x1 (blue) because the former plot lies consistently to the right
of the latter.
Because the shapes of the two distributions aren't
exactly the same, some might say a two-sample Wilcoxon test is
not entirely appropriate. I would not want to claim the Wilcoxon test
finds that the median of x2 to be larger than the median of x1,
but the Wilcox test does find a difference between the two samples.
wilcox.test(x1, x2)

         Wilcoxon rank sum test with continuity correction

data:  x1 and x2
W = 372900, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0

