I am trying to model the distribution of some non normal data, to do so i am fitting many different distributions(Student, Pareto...) to the data. When computing the Kolmogorov Smirnov Statistic for each of the candidate distributions, Can I use the different p values to rank the candidates? Otherwise, how can I use the Kolmogorov Smirnov Test to select the best candidate?

I have found an article which proposes a similar procedure on Page 11 (Chapter 6), but I dont know about the reliability since it is discussed nowhere besides this article : https://www.iiap.res.in/astrostat/School10/LecFiles/Karandikar_Babu_ModelSelGOF_notes.pdf?fbclid=IwAR39zkhXRUD-j4JvCttBr6JOErxkp7h9Ct_Osz7BaXJkgF9wmWtFY4B2w14


  • 2
    $\begingroup$ Why do you feel you need to know the distribution? $\endgroup$ – mdewey Aug 9 at 15:14
  • $\begingroup$ What is your sample size? You are probably better off with visualization methods as a start. Try qqplots, ... search this site! $\endgroup$ – kjetil b halvorsen Aug 9 at 19:36
  • $\begingroup$ I have already tried different methods : qqplot, bic ,wasserstein distance, but I am curious as to how one could use the Kolmogorov Smirnov Test quantile to select the best model out of a large number of candidates, is it theoretically acceptable to compare the p values? My sample size is 2500. $\endgroup$ – idk Aug 9 at 21:01

I wouldn't use the p-value to choose; I'd use the test statistic itself.

More specifically, I'd be quite unlikely to do something like this in the first place, and if I did I'd be quite unlikely to choose the Kolmogorov-Smirnov to do it wth, but if for some reason I did do it, I'd be looking at the actual measure of discrepancy rather than its p-value, since that at least says something about the fit.

Note that this would only make sense to use for comparison if the distributions were fully specified (or otherwise at least if number of parameters were the same for each distribution).

In the case where different models have different numbers of free parameters, I would not expect there to be a good way to compare the discrepancies (nor indeed the p-values) across models using the Kolmogorov-Smirnov distance.

| cite | improve this answer | |
  • $\begingroup$ Wouldnt the use of the quantile of the Kolmogorov Smirnov Distance(which we could bootstrap to get) make more sense since,otherwise, we can be vulnerable to outliers? And what other less known methods for distribution selection can I explore ? Thanks. $\endgroup$ – idk Aug 10 at 7:02
  • $\begingroup$ The statistic is looking at the difference in terms of a vertical difference in cdf. How much effect can an outlier have? The second question is probably better asked as a new question, but consider that (a) the best answer may usually be "don't select a distribution at all" and (b) such a question has already been answered on site a number of times already. $\endgroup$ – Glen_b Aug 10 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.