We know that the original KS-test has a limitation, requiring the Null hypothesis testing underlying distribution to be fully specified, rather than estimated. But in practice, we usually need to test goodness of fit for the fitted distribution.
It is suggested to conduct a Monte Carlo Simulation by Wikipedia for KS-test. But there are very few limited literature review about such simulation. There are several ideas come up in my mind.
Approach 1
It seems that the reason why original KS-test is no longer valid for testing estimated distribution, is that we need a smaller critical threshold KS Distance (d value) to reject the null hypothesis underlying distribution, as it is fitted around the data.
The Lilliefors-test solves such problem. Although it is only for Normal distribution, we can extend its method.
- Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution, like original KS-test.
- Generate n data points by the estimated underlying distribution for M times (around millions), find their KS Distance $D_{m}$ set = {$D_{1}$, $D_{2}$, $D_{3}$, ..., $D_{M}$} for each time.
- Compare the $D_{0}$ value and the percentile of $D_{m}$ set as the p-value.
$P(>D_{0}|Null Hypothesis Distribution)$
Approach 2
Some post said that the original KS-test can not be applied to estimated distribution, because it does not take into account the standard error of estimation (or fitting).
So they suggest bootstrap from raw data as a simulation, but they have not described the details. I think the approach can be as follow.
- Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution A, like original KS-test.
- Bootstrap from raw data and fit distribution again, getting a new fitted distribution B. Find the KS Distance ($D_{1}$) between this new raw sample and new distribution B.
- Repeat step 2 for M time, getting their KS Distance $D_{m}$ set = {$D_{1}$, $D_{2}$, $D_{3}$, ..., $D_{M}$} for each time.
- Compare the $D_{0}$ value and the percentile of $D_{m}$ set, to see whether it is out of the 95% empirical interval.
It seems that this takes into account the estimated error (or confidence interval) about the estimation, which affecting the original KS-test. But it seems that this approach does not follow the p-value definition, or hypothesis definition, assuming Null hypothesis and find the sample statistics probability.
Approach 3
My fellow also comes up an idea, like the approach 2.
- Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution A, like original KS-test.
- Set the Null hypothesis that the data is from the distribution in question (A).
- Generate n points from the distribution in question (A) and fit the same distribution to this new sample which gives parameter set B. Measure the KS distance $D_{1}$ between this sample and the distribution in question with parameter set B.
- Repeat step 3 a large number of times (M) which gives a distribution of KS distances $D_{m}$ assuming the actual data is from the distribution in question using our fitting method.
- If distance $D_{0}$ is greater than 95% of the distances calculated in 4 then reject the Null hypothesis that the data is from the distribution in question with parameter set A.
It seems that it is something parametric bootstrap from approach 2. But it appear strange to generate such sample and fit data again. And it seems still not following the hypothesis definition.
Do you have any idea about these three approaches? or any modification?
Or do you find any formal literature saying simulation details for KS-test for estimated distribution?
I only find Durbin (1973) with details about this. But sorry I can not find the full text about this book. Does anyone have any other electronic version of similar details about simulation KS-test?
Does Anderson-Darling test also suffer from this estimated parameter issue?
Any ideas or discussion are highly appreciated.
Thank you.
