I am using
scipy's ks_samp in order to apply the Kolmogorov-Smirnov-test.
The data I use is twofold:
- I have a dataset
d1which is an evaluation-metric applied on the forecast of a machine-learning model
m1(namely the MASE - Mean Average Scaled Error). These are around 6.000 data points meaning the MASE-result of 6.000 forecasts using
- My second dataset
d2is analogous to
d1with the difference that I used a second model
m2, which slightly differs from
The distribution of both datasets looks like:
As can be seen, the distribution looks pretty much alike. I wanted to underline this fact with a Kolmogorov-Smirnov test. However, the results I get applying
k2_samp indicate the contrary:
from scipy.stats import ks_2samp k2_samp(d1, d2) # Ks_2sampResult(statistic=0.04779414731236298, pvalue=3.8802872942682265e-10)
As I understand, such a pvalue indicates that the distribution is not alike (rejection of H0). But as can be seen on the images it definitely should.
- Am I misunderstand the usage of Kolmogorov-Smirnov and this test is not applicable for the use-case/kind of distribution?
- If first can be answered with yes, what alternative do I have?