0
$\begingroup$

WARNING : MAJOR UPDATE BELOW, I've made a mistake

I am currently working on validating a predictive model that was made some years ago. In doing so, I am comparing the distribution of the Target variable vs Model variables from the old database vs new data.

I did some ploting and also performed a two sample KS-test and I cannot understand the results obtained.

enter image description here enter image description here

Those are two examples of distributions. Where :

  • Param: Is the KS test parameter obtained
  • pval: Is the p-value of the test

I am working in Python and used the stats.ks_2samp(data1,data2) for comparing.

According to documentation from SciPy 'If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.'

Having p-values with zeros up to the 5th decimal place means that we can reject the hypothesis of the two samples coming from the same distributions, but when I look at the plots they look extremely similar.

What I am missing?

If any additional information is needed for further understanding the problem, please let me know.

edits:

Sample sizes

For the graph with param: 0.4576 n: 9363 and m: 17972 For the graph with param: .4687 n: 1455 and m:1923

The variables in the model are clustering variables, so what I am doing is for each variable and each cluster (for example for people aged between 18 and 25, 26-35, 36-45, etc) I see the distribution (to verify that the new clusters still behave like the original ones, that is why I have different sizes for the plots)

I have also tried "capping" the sample size to the minimum of the two, so that m=n, but results were not better either.

Edit 2 : Major update

So, this is a little embarrasing. While I was trying to do the cumulative distribution plot Seaborn wouldn't plot and I had error warnings. So I finally realised that I made a mistake earlier in the code. So.. this are the updated plots with the corresponding cumulatives

enter image description hereenter image description here enter image description hereenter image description here Now in the second row we can see some good improvement.

But there are still some other clusters with the behaviour that originated this post, like: enter image description hereenter image description here

So, eventhough I had made a mistake in the first version, all the answers are still valid because the problem keeps appearing on different distributions.

Updated to keep the question useful for everyone.

Thanks a lot to everyone

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ How many data points do you have? It is a general weakness of all NHST approaches that even trivial departures from the null hypothesis (here: that both distributions are identical) become statistically (!) significant if the sample size is large enough. $\endgroup$
    – Stephan Kolassa
    Commented Sep 24, 2019 at 14:52
  • $\begingroup$ Perhaps the value of the K-S statistic is useful. Nevertheless, you may take a look at a related post about statistical distance. $\endgroup$
    – Ertxiem - reinstate Monica
    Commented Sep 24, 2019 at 15:10
  • $\begingroup$ info added @StephanKolassa $\endgroup$
    – ShallowNC
    Commented Sep 24, 2019 at 15:20

2 Answers 2

Reset to default
5
$\begingroup$

You're right. They look very similar. If the plots you're showing were for the populations, you'd say that they're different. If you knew that two normal populations had means of $0$ and $0.00001$, respectively, you say that they're different. You might have a hard time detecting that difference, but the populations are indeed different.

What's going on is that you appear to have a large enough sample size to detect a subtle difference that may or may not matter to you. However, your test is giving emphatic evidence that the populations are not identical. Whether or not that subtle difference is important to your work is for you to decide.

Building on Stephan Kolassa's comment, statistical significance are practical significance are not the same. As sample sizes get very large, you increase your sensitivity to trivial departures from the null hypothesis. Nonetheless, the test is suggesting that the null is false.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you a lot, very useful. So given large enough samples that sensitivity becomes "tricky" in some way. Is there any way to know (mathematically or a rule of thumb) what is "large enough" ? $\endgroup$
    – ShallowNC
    Commented Sep 24, 2019 at 15:35
  • $\begingroup$ What is "large enough" will depend on what you are doing your analysis for. In some cases, tiny effects are important. In others, small effects are unimportant. You need to invest some subject matter knowledge at this point. $\endgroup$
    – Stephan Kolassa
    Commented Sep 24, 2019 at 16:00
0
$\begingroup$

I agree with Dave's answer. Would it be possible to see a comparison of the cumulative density functions for the probability densities you have shown? The pdfs do look remarkably similar for a ks metric (= max absolute difference between CDFs) of 0.45. Just to rule out other issues.

Perhaps you could consider alternate tests for easier interpretation? How about Cohen's d? This has a convenient table for interpretation (see https://en.wikipedia.org/wiki/Effect_size#Cohen's_d).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.