3
$\begingroup$

I've been redirected here from another forum...

I have used a two sample Kolmogorov-Smirnov test to compare the distributions of two sets of data. Basically I am comparing the error distributions between two measurements when an intervention is made, to determine whether the intervention (a changing measurement parameter) significantly changes the error distribution of the measurement.

I know that the K-S test is a non parametric test, however the distributions of data I'm comparing has turned out to be normally distributed... I know there is probably a number of tests that could be used to compare normally distributed data, but is there a reason not to use the K-S test? Are there any disadvantages (with regard to type1 and 2 errors perhaps)? Is it ok to use it?

I've sort of gone down this route with my data analysis, but the question has come up: why use a non parametric test to compare parametric data? Hopefully K-S is unconventional rather than completely wrong.

Improve this question
$\endgroup$
3
  • $\begingroup$ when you say "error distributions" -- are these some form of residuals from some model-fit? If so, there's a couple of reasons why the Kolmogorov-Smirnov statistic won't have the tabulated distribution. $\endgroup$
    – Glen_b
    Commented Aug 14, 2014 at 13:37
  • $\begingroup$ no not residuals. essentially its a length measurement using an optical system. I take around 500 measurements and get a distribution of the error in the length measurement. I then change a parameter in my measuring system, take another 500 measurements and get a distribution of the error for that. I then compare the two distributions to see if they are significantly different using the K-S test. Its essentially seeing if the measurement system is as accurate when a parameter in the process is changed. $\endgroup$
    – Rob
    Commented Aug 14, 2014 at 13:57
  • $\begingroup$ Thanks for making that clear. The KS test should be fine, but if you know your data will actually be be normally distributed, its not as efficient as it could be at picking differences. It's not clear how you could know this though (approximately normal, perhaps). $\endgroup$
    – Glen_b
    Commented Aug 14, 2014 at 20:30

2 Answers 2

Reset to default
2
$\begingroup$

There is nothing wrong with using the K-S test. The reason you might want to use other tests such as ones tailored from a Normal distribution is that these tests may give you more power. But if you are not certain of normality, I think K-S test will do just fine. I personally like the K-S test because it has fewer assumptions compared to other tests. Back in the day when we didn't have the computing power that we do now, parametric test were easier computationally and valid if they fit the assumptions. But now with the computing power we have, there isn't to much of a reason not to do nonparametric tests

The one thing you will want to make sure is that the you have a large enough of sample size for both to ensure empirical CDF is close to actual CDF.

Improve this answer
$\endgroup$
1
$\begingroup$

I think K-S test has an important limitation which requires the underlying distribution must be fully defined. That is why if mean and variance parameters are estimated from the actual data, K-S test are no longer valid and its critical values are difficult to get. You can have a look at some papers writing about this problem: Koul and Swordson (2011), Bai(2003)...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.