# Simulation of KS-test with estimated parameters

We know that the original KS-test has a limitation, requiring the Null hypothesis testing underlying distribution to be fully specified, rather than estimated. But in practice, we usually need to test goodness of fit for the fitted distribution.

It is suggested to conduct a Monte Carlo Simulation by Wikipedia for KS-test. But there are very few limited literature review about such simulation. There are several ideas come up in my mind.

## Approach 1

It seems that the reason why original KS-test is no longer valid for testing estimated distribution, is that we need a smaller critical threshold KS Distance (d value) to reject the null hypothesis underlying distribution, as it is fitted around the data.

The Lilliefors-test solves such problem. Although it is only for Normal distribution, we can extend its method.

1. Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution, like original KS-test.
2. Generate n data points by the estimated underlying distribution for M times (around millions), find their KS Distance $D_{m}$ set = {$D_{1}$, $D_{2}$, $D_{3}$, ..., $D_{M}$} for each time.
3. Compare the $D_{0}$ value and the percentile of $D_{m}$ set as the p-value.

$P(>D_{0}|Null Hypothesis Distribution)$

## Approach 2

Some post said that the original KS-test can not be applied to estimated distribution, because it does not take into account the standard error of estimation (or fitting).

So they suggest bootstrap from raw data as a simulation, but they have not described the details. I think the approach can be as follow.

1. Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution A, like original KS-test.
2. Bootstrap from raw data and fit distribution again, getting a new fitted distribution B. Find the KS Distance ($D_{1}$) between this new raw sample and new distribution B.
3. Repeat step 2 for M time, getting their KS Distance $D_{m}$ set = {$D_{1}$, $D_{2}$, $D_{3}$, ..., $D_{M}$} for each time.
4. Compare the $D_{0}$ value and the percentile of $D_{m}$ set, to see whether it is out of the 95% empirical interval.

It seems that this takes into account the estimated error (or confidence interval) about the estimation, which affecting the original KS-test. But it seems that this approach does not follow the p-value definition, or hypothesis definition, assuming Null hypothesis and find the sample statistics probability.

## Approach 3

My fellow also comes up an idea, like the approach 2.

1. Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution A, like original KS-test.
2. Set the Null hypothesis that the data is from the distribution in question (A).
3. Generate n points from the distribution in question (A) and fit the same distribution to this new sample which gives parameter set B. Measure the KS distance $D_{1}$ between this sample and the distribution in question with parameter set B.
4. Repeat step 3 a large number of times (M) which gives a distribution of KS distances $D_{m}$ assuming the actual data is from the distribution in question using our fitting method.
5. If distance $D_{0}$ is greater than 95% of the distances calculated in 4 then reject the Null hypothesis that the data is from the distribution in question with parameter set A.

It seems that it is something parametric bootstrap from approach 2. But it appear strange to generate such sample and fit data again. And it seems still not following the hypothesis definition.

Do you have any idea about these three approaches? or any modification?

Or do you find any formal literature saying simulation details for KS-test for estimated distribution?

Does Anderson-Darling test also suffer from this estimated parameter issue?

Any ideas or discussion are highly appreciated.

Thank you.

• Wikipedia is talking about your approach 1, the Lilliefors test approach. You might be interested to learn that Lilliefors' original simulation size was 1000, I think for both the normal and exponential cases (but this is in the 1960s - we can do hundreds of thousands, even millions in pretty quick time). – Glen_b Aug 13 '14 at 10:53
• @Glen_b Yea, the approach 1 is extended from Wikipedia Lilliefors test from my understanding into different distribution. Just wonder whether my understanding or extension is correct. Or other approach is more correct? Thank you. – Vincent Aug 13 '14 at 10:59
• I'll write a brief answer. – Glen_b Aug 13 '14 at 12:41

It is a Lilliefors test, and your first and third items are pretty close to how to do it.

The statement that Lilliefors test is only for the normal distribution is wrong. He did one for the exponential as well (you can even see that in the references at the bottom of the Wikipedia page you linked to), and the technique should work with quite a few other distributions.

However, I don't think you quite have it right.

There are a couple of ways of organizing it, but one approach that works for continuous distributions is as follows:

Repeat many times:

1. Simulate a sample of the desired sample size from the assumed distribution.

2. Estimate the parameters of the distribution.

3. Treating the estimated parameters as the population values, transform to uniformity via the probability integral transform. (You can compute a KS statistic without transforming at this step; however, it makes the computation a bit simpler.)

4. Compute a KS test statistic.

Collect the simulated statistics, and work out the proportion of times the simulated statistic is at least as extreme (more consistent with $H_1$) as the observed sample value.

If you have it right you should be able to reproduce the results for Lilliefors paper (to the limited accuracy he had, anyway.

• This method seems my Approach 3, right? Thank you. – Vincent Aug 13 '14 at 13:32
• This method seems my Approach 3, right? Approach 1 only generate new samples and compare to the original underlying distribution, but not fitting new distribution. Thank you. – Vincent Aug 13 '14 at 13:39
• Sorry, I was going by which one you called "Lilliefors" -- but no, it's not exactly your approach 3 either. – Glen_b Aug 13 '14 at 21:21
• What exactly it means to "work out the proportion of times the simulated statistic is at least as extreme (more consistent with H1H1) as the observed sample value"? With this we will obtain a new p-value that we can compare against our significance level? I am not sure I understood how the last part is computed. – Cesar Sep 28 '16 at 12:20
• You seem to have a number of things you want to know about. Perhaps you could consider asking a new question? You can always link to the question here for background. – Glen_b Sep 29 '16 at 10:38