WARNING : MAJOR UPDATE BELOW, I've made a mistake
I am currently working on validating a predictive model that was made some years ago. In doing so, I am comparing the distribution of the Target variable vs Model variables from the old database vs new data.
I did some ploting and also performed a two sample KS-test and I cannot understand the results obtained.
Those are two examples of distributions. Where :
- Param: Is the KS test parameter obtained
- pval: Is the p-value of the test
I am working in Python and used the stats.ks_2samp(data1,data2) for comparing.
According to documentation from SciPy 'If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.'
Having p-values with zeros up to the 5th decimal place means that we can reject the hypothesis of the two samples coming from the same distributions, but when I look at the plots they look extremely similar.
What I am missing?
If any additional information is needed for further understanding the problem, please let me know.
For the graph with param: 0.4576 n: 9363 and m: 17972 For the graph with param: .4687 n: 1455 and m:1923
The variables in the model are clustering variables, so what I am doing is for each variable and each cluster (for example for people aged between 18 and 25, 26-35, 36-45, etc) I see the distribution (to verify that the new clusters still behave like the original ones, that is why I have different sizes for the plots)
I have also tried "capping" the sample size to the minimum of the two, so that m=n, but results were not better either.
Edit 2 : Major update
So, this is a little embarrasing. While I was trying to do the cumulative distribution plot Seaborn wouldn't plot and I had error warnings. So I finally realised that I made a mistake earlier in the code. So.. this are the updated plots with the corresponding cumulatives
So, eventhough I had made a mistake in the first version, all the answers are still valid because the problem keeps appearing on different distributions.
Updated to keep the question useful for everyone.
Thanks a lot to everyone