2
$\begingroup$

I notice there are a lot of questions on the topic (most are along the lines of: how to do a Chi-squared goodness-of-fit test), but mine is perhaps more general.

Using an example I recently encountered. I have a one-sample dataset, where the data are strictly positive, and have a right-skewed distribution. I make a guess that the data may be fitted to one of the four distributions:

  1. Log-normal
  2. Exponential
  3. Weibull
  4. Gamma

At first, I assess the distribution visually using geom_histotgram (from package ggplot2) and plot the estimated distributions using stat_function, where the distribution parameters are estimated using fitdistr (from package MASS).

ggplot(DT4, aes(TTE)) + geom_histogram(aes(y=..density..), bins=90) +
  stat_function(fun=dlnorm,args=fitdistr(DT4[, TTE],"log-normal")$estimate, color=pal[1], size=.5) +
  stat_function(fun=dexp,args=fitdistr(DT4[, TTE],"exponential")$estimate, color=pal[2], size=.5) +
  stat_function(fun=dweibull,args=fitdistr(DT4[, TTE],"weibull")$estimate, color=pal[3], size=.5) +
  stat_function(fun=dgamma,args=fitdistr(DT4[, TTE],"gamma")$estimate, color=pal[4], size=.5) +
  annotate("text", label="Log-normal", x=250, y=0.025, color=pal[1]) +
  annotate("text", label="Exponential", x=250, y=0.023, color=pal[2]) +
  annotate("text", label="Weibull", x=250, y=0.021, color=pal[3]) +
  annotate("text", label="Gamma", x=250, y=0.019, color=pal[4])

enter image description here

All four distributions have a faily good fit, with log-normal perhaps having the best fit, just from a visual assessment.

However, imagine you are a professional statistician (which I am not) who wants to assess the fit rigorously. I have looked around and found numerous goodness-of-fit tests, including

  1. Chi-squared / G-test (designed for discrete data, but may be applied to continuous data if it is divided into arbitrary bins)
  2. Kolmogorov-Smirnov
  3. Cramér–von Mises
  4. Anderson-Darling
  5. Maximum Likelihood Estimate
  6. Information Criteria: AIC/AIC$_\text{c}$/CAIC/BIC/HQIC
  7. Any others I have missed?

Question

  1. How do we choose which goodness-of-fit to use to compare the fit of different distributions? In straightforward cases, one distribution may have the best fit according to every test across the board, but in more marginal cases, what if one distribution is best according to one test, and another is best according to another test? (Is there a reference, like a book chapter, that explores this question?)

  2. Is there a go-to R function/package that conveniently performs a range of goodness-of-fit tests all at once?

$\endgroup$
  • 1
    $\begingroup$ The professional will protect herself from errors by (among other things) adjusting for the mistake of selecting the procedure based on a preliminary examination of the data (often that's a minor issue) and by not conducting myriad tests and selecting among their results (doing so would be a major mistake). How one does this, and the principles that inform the process, are the subject of basic (but not necessary elementary!) textbooks of statistics. $\endgroup$ – whuber Jan 14 at 14:17
  • $\begingroup$ @whuber would you recomment one such book, ideally with a reference to a chapter? $\endgroup$ – Mihael Jan 14 at 15:06
6
$\begingroup$

Great question! (Almost a professional statistician here!) Let's see if I can try to answer your question. Whuber gave a comment on what the typical strategy that someone might follow. I'll try to complement that by providing a couple papers that talk specifically about properties of tests when assumptions are violated. I'm doing this since, if I had a choice between multiple goodness of fit tests, I would do a process of elimination by saying which tests are less "valid" given my data and potential sources of noise.

When it comes to goodness of fit tests, there a few different criteria to keep in mind, and assumptions on the estimated distributions tend to matter - the degree to which they matter (i.e., robustness to assumption violations varies). Some of the papers below talk about validity and goodness of fit under different assumption violations.

Estimating Parameters in Kolmogorov Smirnov Test

On the Kolmogorov Smirnoff test, for example, in the empirical-to-theoretical distribution comparison, if you have to estimate parameters to conduct the test, then the p-values are technically not valid. The paper below gives a quick review on this.

Lilliefors, H. W. (1969). On the Kolmogorov-Smirnov test for the exponential distribution with mean unknown. Journal of the American Statistical Association, 64(325), 387-389. https://amstat.tandfonline.com/doi/pdf/10.1080/01621459.1969.10500983?needAccess=true#.Xh3UTFNKhQI

Chi-Square Test with Heterogeneous variance

Usually in regression, we assume constant variance. Sometimes, this isn't the case, but there are robust alternatives to the standard chi-square. This paper talks about a statistic that is asympototically chi-square but is robust to non-constant variance as well.

Lipsitz, Stuart R., and John F. Buoncristiani. "A robust goodness‐of‐fit test statistic with application to ordinal regression models." Statistics in medicine 13, no. 2 (1994): 143-152. https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.4780130205

Using Mahalonobis Distance to mitigate against Outliers

Cerioli, Andrea, Alessio Farcomeni, and Marco Riani. "Robust distances for outlier-free goodness-of-fit testing." Computational Statistics & Data Analysis 65 (2013): 29-45. https://www.sciencedirect.com/science/article/pii/S016794731200134X

Goodness of Fit R package

Here are a couple of packages that perform specific goodness of fit tests. I don't know if there exists one that does them all at once? I will keep looking and see if I can find one.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Upvoted for the excellence of the answer. $\endgroup$ – James Phillips Jan 14 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.