14
$\begingroup$

I often hear the statement that the KS-Test (for comparing two distributions described by two samples) is too strict if the sample size is rather large, meaning that the 0-hypothesis of equal distributions is rejected too often.

In my application I am given 2 samples of 1500 observations each and a low P-value for the given KS-statistic. Neverthless my data looks similar if I plot histograms/density estimates.

My question: can we make the statement KS-test being too strict more rigorous (can we state a threshold for the number of observations?). Is it "true" at all? Is there any reference?

Thank you! PS: I know this looks like a dublicate but checking the first suggestions by the site I didn't see a clear dublicate. If this is a dublicate please refer me to it ! :)

EDIT: I add some background. The distributions that I want to compare are explanatory variables in a logistic regression model. It was developed on some sample (A) and I want to apply it to another sample (B). I can not test the model as I can don't know the outcome on B. One approach that people use here to assess whether the model gives reasonable results on B is to test the variables for similar distributions (KS-test, Chi-squared).

Is there a better approach to assess whether it makes statistically sense (speaking of discriminatory power) to apply a model developed on A to B without knowing the results on B?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ "too strict" in itself does not mean anything. "too strict" to achieve what? It is certainly to strict for what many people try to use it, but a null hypothesis test can not be too strict in itself. You need to define what you want from it and then you can try to determine, if it is too strict for that particular purpose. $\endgroup$
    – Bernhard
    Commented Mar 16, 2018 at 12:57
  • 2
    $\begingroup$ @Bernhard you are right of course. It does not make sense but people in practice use this way to express a mismatch between the test and their expectation. The aim of this question is to explore ways to formulate this. $\endgroup$
    – Richi W
    Commented Mar 16, 2018 at 14:50

1 Answer 1

Reset to default
23
$\begingroup$

With a test you try to find deviations from the Null hypothesis. The larger the sample the better we are at detecting such deviations, even trivially small ones. So if you do test in large samples you will reject the null hypothesis quite often due to substantively trivial deviations. This is what many people mean when they say that statistical tests reject the null too often in large samples.

Strictly speaking they are wrong: the test correctly answered the question the user posed, it is just that the question the user posed was not the question (s)he wanted to ask... But try devise a testing procedure for the hypothesis: Two distributions are equal ignoring substantively trivial differences. We (humans) can decide what is substantively trivial, a procedure like statistical testing cannot.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for this answer. I will edit my questin (add a part) and maybe you have an idea what to do in this setting. $\endgroup$
    – Richi W
    Commented Mar 16, 2018 at 14:51
  • $\begingroup$ My edit to the question is meant to clarify which difference of distributions would be "too much": if the discriminatory power is reduced (which can be made more rigorous) when the model is applied to different data ... $\endgroup$
    – Richi W
    Commented Mar 16, 2018 at 15:49
  • 1
    $\begingroup$ I'm currently dealing with a very similar problem, and it is indeed a problem with the null hypothesis. The null for the KS test is that the two samples came from the exact same distribution. In reality you may expect that there are sources of noise or batch effects between the two samples that you will not be able to completely remove, and with enough samples the KS test will pick up on this. $\endgroup$
    – JaredL
    Commented Mar 16, 2018 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.