# Two-sample KS test results vs distributions

WARNING : MAJOR UPDATE BELOW, I've made a mistake

I am currently working on validating a predictive model that was made some years ago. In doing so, I am comparing the distribution of the Target variable vs Model variables from the old database vs new data.

I did some ploting and also performed a two sample KS-test and I cannot understand the results obtained.

Those are two examples of distributions. Where :

• Param: Is the KS test parameter obtained
• pval: Is the p-value of the test

I am working in Python and used the stats.ks_2samp(data1,data2) for comparing.

According to documentation from SciPy 'If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.'

Having p-values with zeros up to the 5th decimal place means that we can reject the hypothesis of the two samples coming from the same distributions, but when I look at the plots they look extremely similar.

What I am missing?

If any additional information is needed for further understanding the problem, please let me know.

# edits:

## Sample sizes

For the graph with param: 0.4576 n: 9363 and m: 17972 For the graph with param: .4687 n: 1455 and m:1923

The variables in the model are clustering variables, so what I am doing is for each variable and each cluster (for example for people aged between 18 and 25, 26-35, 36-45, etc) I see the distribution (to verify that the new clusters still behave like the original ones, that is why I have different sizes for the plots)

I have also tried "capping" the sample size to the minimum of the two, so that m=n, but results were not better either.

## Edit 2 : Major update

So, this is a little embarrasing. While I was trying to do the cumulative distribution plot Seaborn wouldn't plot and I had error warnings. So I finally realised that I made a mistake earlier in the code. So.. this are the updated plots with the corresponding cumulatives

Now in the second row we can see some good improvement.

But there are still some other clusters with the behaviour that originated this post, like:

So, eventhough I had made a mistake in the first version, all the answers are still valid because the problem keeps appearing on different distributions.

Updated to keep the question useful for everyone.

Thanks a lot to everyone

• How many data points do you have? It is a general weakness of all NHST approaches that even trivial departures from the null hypothesis (here: that both distributions are identical) become statistically (!) significant if the sample size is large enough. – Stephan Kolassa Sep 24 '19 at 14:52
• Perhaps the value of the K-S statistic is useful. Nevertheless, you may take a look at a related post about statistical distance. – Ertxiem - reinstate Monica Sep 24 '19 at 15:10
• info added @StephanKolassa – ShallowNC Sep 24 '19 at 15:20

You're right. They look very similar. If the plots you're showing were for the populations, you'd say that they're different. If you knew that two normal populations had means of $$0$$ and $$0.00001$$, respectively, you say that they're different. You might have a hard time detecting that difference, but the populations are indeed different.