# Is the Kolmogorov-Smirnov-Test too strict if the sample size is large?

I often hear the statement that the KS-Test (for comparing two distributions described by two samples) is too strict if the sample size is rather large, meaning that the 0-hypothesis of equal distributions is rejected too often.

In my application I am given 2 samples of 1500 observations each and a low P-value for the given KS-statistic. Neverthless my data looks similar if I plot histograms/density estimates.

My question: can we make the statement KS-test being too strict more rigorous (can we state a threshold for the number of observations?). Is it "true" at all? Is there any reference?

Thank you! PS: I know this looks like a dublicate but checking the first suggestions by the site I didn't see a clear dublicate. If this is a dublicate please refer me to it ! :)

EDIT: I add some background. The distributions that I want to compare are explanatory variables in a logistic regression model. It was developed on some sample (A) and I want to apply it to another sample (B). I can not test the model as I can don't know the outcome on B. One approach that people use here to assess whether the model gives reasonable results on B is to test the variables for similar distributions (KS-test, Chi-squared).

Is there a better approach to assess whether it makes statistically sense (speaking of discriminatory power) to apply a model developed on A to B without knowing the results on B?

• "too strict" in itself does not mean anything. "too strict" to achieve what? It is certainly to strict for what many people try to use it, but a null hypothesis test can not be too strict in itself. You need to define what you want from it and then you can try to determine, if it is too strict for that particular purpose. Mar 16, 2018 at 12:57
• @Bernhard you are right of course. It does not make sense but people in practice use this way to express a mismatch between the test and their expectation. The aim of this question is to explore ways to formulate this. Mar 16, 2018 at 14:50