15
$\begingroup$

I've read that Kolmogorov-Smirnov test should not be used to test the goodness of fit of a distribution whose parameters have been estimated from the sample.

Does make sense to split my sample in two and use the first half for parameter estimation and the second one for KS-test?

Thanks in advance

$\endgroup$
  • 1
    $\begingroup$ What distribution do you want to test against & why? $\endgroup$ – gung - Reinstate Monica Dec 3 '12 at 21:56
  • $\begingroup$ I suspect the data follows a exponential distribution. $\endgroup$ – sortega Dec 4 '12 at 8:24
15
$\begingroup$

The better approach is to compute your critical value of p-value by simulation. The problem is that when you estimate the parameters from the data rather than using hypothesized values then the distribution of the KS statistic does not follow the null distribution.

You can instead ignore the p-values from the KS test and instead simulate a bunch of datasets from the candidate distribution (with a meaningful set of parameters) of the same size as your real data. Then for each set estimate the parameters and do the KS test using the estimated parameters. You p-value will be the proportion of test statistics from the simulated sets that are more extreeme than for your original data.

Added Example

Here is an example using R (hopefully readable/understandable for people who use other programs).

A simple example using the Normal distribution as the null hypothesis:

tmpfun <- function(x, m=0, s=1, sim=TRUE) {
  if(sim) {
    tmp.x <- rnorm(length(x), m, s)
  } else {
    tmp.x <- x
  }
  obs.mean <- mean(tmp.x)
  obs.sd <- sd(tmp.x)
  ks.test(tmp.x, 'pnorm', mean=obs.mean, sd=obs.sd)$statistic
}

set.seed(20200319)
x <- rnorm(25, 100, 5)

out <- replicate(1000, tmpfun(x))

hist(out)
abline(v=tmpfun(x, sim=FALSE))
mean(out >= tmpfun(x, sim=FALSE))

The function will either compute the KS test statistic from the actual data (sim=FALSE) or simulate a new dataset of the same size from a normal distribution with specified mean and sd. Then in either case will compute the test statistic comparing to a normal distribution with the same mean and sd as the sample (original or simulated).

The code then runs 1,000 simulations (feel free to change and rerun) to get/approximate the distribution of the test statistic under the NULL (but with estimated parameters) then finally compares the test statistic for the original data to this NULL distribution.

We can simulate the whole process (simulations within simulations) to see how it compares to the default p-values:

tmpfun2 <- function(B=1000) {
  x <- rnorm(25, 100, 5)
  out <- replicate(B, tmpfun(x))
  p1 <- mean(out >= tmpfun(x, sim=FALSE))
  p2 <- ks.test(x, 'pnorm', mean=mean(x), sd=sd(x))$p.value
  return(c(p1=p1, p2=p2))
}

out <- replicate(1000, tmpfun2())

par(mfrow=c(2,1))
hist(out[1,])
hist(out[2,])

For my simulation, the histogram of the simulation based p-values is fairly uniform (which is should be since the NULL is true), but the p-values for the ks.test function are bunched up much more against 1.0.

You can change anything in the simulations to estimate power by having the original data come from a different distribution, or using a different Null distribution, etc. The normal is probably the simplest since the mean and variance are independent, more tuning may be needed for other distributions.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ I find the solution a little confusing (at least for me); what do you mean by "a meaningful set of parameters" for the candidate distribution? You initially don't know the parameters of the candidate distribution, how would you know what a "meaningful set of parameters" is? $\endgroup$ – Néstor Dec 4 '12 at 1:05
  • $\begingroup$ You can try different sets of parameters to see if it makes a difference or not (for the normal it does not, but some distributions may). Then think about the science behind your data, or talk to a expert in the area, you should be able to get a general idea where to start, e.g. I have know idea what the average height of adult males is in Nigeria, but I am pretty certain that it is positive and less than 3 meters. $\endgroup$ – Greg Snow Dec 5 '12 at 20:04
  • $\begingroup$ @GregSnow I came across this post as it is relevant to my current work. I was wondering if there is any theoretical justification for the method you suggest? That is, how do we know that the proposed "p-value" is indeed uniformly distributed from 0 to 1? The proposed p-value does nto seem to be the conventional p-value because the Null hypothesis is now a set of distributions $\endgroup$ – renrenthehamster Jun 24 '14 at 15:06
  • 1
    $\begingroup$ @LilyLong, simulations used to be much more difficult and time consuming, so the tests were developed to be quicker/easier than simulation, some of the early tables were created by simulation. Many tests can now easily be replaced by simulation, but will probably be with us for a while longer due to tradition and simplicity. $\endgroup$ – Greg Snow Nov 20 '17 at 18:10
  • 1
    $\begingroup$ @Crush_on_You, No ignore all p-values. The simulated p-value is the proportion of simulated test statistics (under the NULL) that are more extreme than the test statistic from your actual data. I have added some example code above to illustrate. $\endgroup$ – Greg Snow Mar 19 at 16:33
8
$\begingroup$

Sample splitting might perhaps reduce the problem with the distribution of the statistic, but it doesn't remove it.

Your idea avoids the issue that the estimates will be 'too close' relative to the population values because they're based on the same sample.

You aren't avoiding the problem that they're still estimates. The distribution of the test statistic is not the tabulated one.

In this case it increases the rejection rate under the null, instead of dramatically reducing it.

A better choice is to use a test where the parameters aren't assumed known, such as a Shapiro Wilk.

If you're wedded to a Kolmogorov-Smirnov type of test, you can take the approach of Lilliefors' test.

That is, to use the KS statistic but have the distribution of the test statistic reflect the effect of parameters estimation - simulate the distribution of the test statistic under parameter estimation. (It's no longer distribution-free, so you need new tables for each distribution.)

http://en.wikipedia.org/wiki/Lilliefors_test

Liliefors used simulation for the normal and the exponential case, but you can easily do it for any specific distribution; in something like R it's a matter of moments to simulate 10,000 or 100,000 samples and get a distribution of the test statistic under the null.

[An alternative might be to consider the Anderson-Darling, which does have the same issue, but which - judging from the book by D'Agostino and Stephens (Goodness-of-fit-techniques) seems to be less sensitive to it. You could adapt the Lilliefors idea, but they suggest a relatively simple adjustment that seems to work fairly well.]

But there are other approaches still; there are families of smooth tests of goodness of fit, for example (e.g. see the book by Rayner and Best) that in a number of specific cases can deal with parameter estimation.

* the effect can still be pretty large - perhaps bigger than would normally be regarded as acceptable; Momo is right to express concern about it. If a higher type I error rate (and a flatter power curve) is a problem, then this may not be an improvement!

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ could you explain how "sample splitting would solve the problem with the distribution of the statistic"? In my opinion, the parameters would be estimated from a subsample and then plugged in for the KS test of the second subsample, but the parameters would still be associated with sampling error that is not accounted for in the null distribution. This sounds to me as if one could with a similar idea split a sample from a normal distribution, estimate standard deviations in one subsample and carry out a mean comparison with the standard normal rather than the t-dist in the second subsample. $\endgroup$ – Momo Dec 3 '12 at 23:36
  • 1
    $\begingroup$ @Momo 'solve' is too strong; 'reduce' is better. If parameters are estimated from the same observations you're testing, then - unless you account for that effect - the deviations of the sample from the distribution will be 'too small' - the rejection rate goes waay down. Using another sample removes that effect. The parameter values resulting from estimating from a second sample still suffer from sampling error. That will have some impact on the test (pushes up the type I error rate), but won't have the dramatic biasing effect that using the same data for both does. $\endgroup$ – Glen_b Dec 4 '12 at 0:01
  • $\begingroup$ @Momo I have edited my comment to remove 'solve' and replace it with some explanation $\endgroup$ – Glen_b Dec 4 '12 at 0:09
6
$\begingroup$

I'm afraid that wouldn't solve the problem. I believe the problem is not that the parameters are estimated from the same sample but from any sample at all. The derivation of the usual null distribution of the KS test does not account for any estimation error in the parameters of the reference distribution, but rather sees them as given. See also Durbin 1973 who discusses this issues at length and offers solutions.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ These are actually two separate problems. If you use the same data to estimate the parameters and to do the KS-Test, you'll generally see inflated p-values, because you essentially adapt the distribution to the data before testing against it. If you use two independent sets of samples,however, this is not the case. However, imprecise parameter estimates might decrease the p-values you get in this case, because now you're essentially testing against a (slightly) wrong distribution. $\endgroup$ – fgp Nov 25 '15 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.