0
$\begingroup$

I compared two sets of data using KS-test. First set is empirical data X1 and the second is expected data X2 which is randomly sampled, normally distributed with mean $\mu$ and std dev $\sigma$.

The length of X2 is $10^6$. In the plot of their CDF, it looks like that both of them have similar distribution. When I did KS-test with X1 length is $10^3$, the result is H=0, which was correct.

However, when I tried KS-test with X1 length > $10^3$, I got wrong results (H=1), even though the plot showed that they belong to the similar distribution. I attached the plot here, when I got the wrong result. For the plot, I used X1 size=$10^6$ and X2 size=$10^6$. Empirical CDF in red, Expected CDF in blue. I used standard Matlab command (kstest2)

Empirical CDF in red, Expected CDF in blue

Is there any opinion related to this issue?

Improve this question
$\endgroup$
12
  • $\begingroup$ Why are you comparing to random normal data rather than comparing to a normal cdf (i.e. why use a two sample test rather than a one sample test)? Where did the values of $\mu$ and $\sigma$ come from? $\endgroup$
    – Glen_b
    Feb 24, 2017 at 9:24
  • $\begingroup$ Sample X1 (empirical) comes from the outcome of my experiment. The PDF looks so much like normal. Then, X2 is normally distributed random number generated with $\mu$ comes from average of X1 and so does $\sigma$ from the stddev of X1. I thought that using KS I can justified that X1 is likely to be normal. $\endgroup$
    – enas
    Feb 24, 2017 at 12:31
  • $\begingroup$ Thanks for clarifying. 1. You have one sample -- so why artificially go to a two sample test instead of the one-sample test that already exists? (i.e. why not test your sample against a population distribution of normality rather than generating a random sample from a normal?) 2. If you're using parameters estimated from the sample, the usual KS test (whether one or two sample) doesn't work (the fit always looks much better to the test than it should because it doesn't take account of the fact that you are fitting your sample, so your p-values are always too big); ctd $\endgroup$
    – Glen_b
    Feb 25, 2017 at 0:11
  • $\begingroup$ ctd... the one-sample version of the KS for estimated parameters is the Lilliefors test . 3. For that matter, why not a more powerful test? A Shapiro-Wilk, say, which also totally avoids the issue of estimating parameters ... 4. No goodness of fit test will allow you to say "X1 is likely to be normal."; If that's your purpose, you're wasting your time with hypothesis tests altogether. $\endgroup$
    – Glen_b
    Feb 25, 2017 at 0:11
  • 1
    $\begingroup$ total amount of deviation. $\endgroup$
    – enas
    Mar 2, 2017 at 6:51

2 Answers 2

Reset to default
1
$\begingroup$

When I did KS-test with X1 length is 103103, the result is H=0, which was correct.

This is incorrect. You have failed to reject your null hypothesis, but that doesn't mean the correct result is H=0. All you can say, at that particular sample size the test is not powerful enough to reject the null hypothesis and conclude your empirical CDFs are statistically different.

However, when I tried KS-test with X1 length > 103103, I got wrong results (H=1)

The results is correct. With more samples, the KS test correctly reject your null hypothesis. It should be rejected because they are similar but not identical.

Practically, I wouldn't even both to run KS test here. It's clear the two distributions are very close, why bother? Statistics is not magic, it can't tell you anything not in your data.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This enlightened me. However, just displaying the two distribution's CDF is justified enough to say that they are identical/similar? I need some quantitative approach to proof the normality of the data. $\endgroup$
    – enas
    Feb 24, 2017 at 12:34
  • $\begingroup$ @enas not always, but your example is just too easy $\endgroup$
    – ABCD
    Feb 24, 2017 at 12:37
  • $\begingroup$ It seems easy visually, but how to show that quantitatively? $\endgroup$
    – enas
    Feb 24, 2017 at 12:39
  • $\begingroup$ @enas KS test. Look at the test statistic and how much sample size you would need to reject, $\endgroup$
    – ABCD
    Feb 24, 2017 at 12:40
0
$\begingroup$

The result of your first test is that you failed to reject the null-hypothesis. This is different from saying that the null-hypothesis is true. Failing to reject means that with the current data you were not able to detect a difference between the two curves. As the sample size gets larger you will be able to detect smaller and smaller deviations. In the graph the two curves are not the same, so no surprise that at some point, as you increased the sample size, you will be able to detect those differences. That does not mean that those deviations are substantively meaningful, that is up to you to decide.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your explanation is very clear to me. I was not too familiar with statistics before. Anyway, how to say that such deviations are substantial of not? is there any quantitative method to show that? $\endgroup$
    – enas
    Feb 24, 2017 at 12:37
  • 1
    $\begingroup$ The graph conveys a lot of information. You cannot condense that into one number without loosing a lot of information. So if you want to have this comparison in one or a small number of numbers you have to choose specific aspects of that distribution, e.g. the variance and skewness. You will run the risk that you will miss important aspects. If you don't want to run that risk, you'll have to stick to the graph. $\endgroup$
    – Maarten Buis
    Feb 24, 2017 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.