# Two-sample KS test results vs distributions

WARNING : MAJOR UPDATE BELOW, I've made a mistake

I am currently working on validating a predictive model that was made some years ago. In doing so, I am comparing the distribution of the Target variable vs Model variables from the old database vs new data.

I did some ploting and also performed a two sample KS-test and I cannot understand the results obtained.

Those are two examples of distributions. Where :

• Param: Is the KS test parameter obtained
• pval: Is the p-value of the test

I am working in Python and used the stats.ks_2samp(data1,data2) for comparing.

According to documentation from SciPy 'If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.'

Having p-values with zeros up to the 5th decimal place means that we can reject the hypothesis of the two samples coming from the same distributions, but when I look at the plots they look extremely similar.

What I am missing?

If any additional information is needed for further understanding the problem, please let me know.

# edits:

## Sample sizes

For the graph with param: 0.4576 n: 9363 and m: 17972 For the graph with param: .4687 n: 1455 and m:1923

The variables in the model are clustering variables, so what I am doing is for each variable and each cluster (for example for people aged between 18 and 25, 26-35, 36-45, etc) I see the distribution (to verify that the new clusters still behave like the original ones, that is why I have different sizes for the plots)

I have also tried "capping" the sample size to the minimum of the two, so that m=n, but results were not better either.

## Edit 2 : Major update

So, this is a little embarrasing. While I was trying to do the cumulative distribution plot Seaborn wouldn't plot and I had error warnings. So I finally realised that I made a mistake earlier in the code. So.. this are the updated plots with the corresponding cumulatives

But there are still some other clusters with the behaviour that originated this post, like:  So, eventhough I had made a mistake in the first version, all the answers are still valid because the problem keeps appearing on different distributions.

Updated to keep the question useful for everyone.

Thanks a lot to everyone

• How many data points do you have? It is a general weakness of all NHST approaches that even trivial departures from the null hypothesis (here: that both distributions are identical) become statistically (!) significant if the sample size is large enough. – Stephan Kolassa Sep 24 '19 at 14:52
• Perhaps the value of the K-S statistic is useful. Nevertheless, you may take a look at a related post about statistical distance. – Ertxiem - reinstate Monica Sep 24 '19 at 15:10
• info added @StephanKolassa – ShallowNC Sep 24 '19 at 15:20

You're right. They look very similar. If the plots you're showing were for the populations, you'd say that they're different. If you knew that two normal populations had means of $$0$$ and $$0.00001$$, respectively, you say that they're different. You might have a hard time detecting that difference, but the populations are indeed different.