20
$\begingroup$

This seems so elementary, but I always get stuck at this point…

Most of the data I deal with are non-normal, and most of the analyses based on a GLM structure. For my current analysis, I have a response variable that is "walking speed" (meters/minute). It's easy for me to identify that I cannot use OLS, but then, I have great uncertainty in deciding what family (Gamma, Weibull, etc.) is appropriate!

I use Stata and look at diagnostics like residuals and heteroscedasticity, residuals vs. fitted values, etc.

I am aware that count data can take the form of a rate (e.g. incidence rates) and have used gamma (the analog to overdispersed discrete negative binomial models), but just would like a "smoking gun" to say YES, YOU HAVE THE RIGHT FAMILY. Is looking at the standardized residuals versus the fitted values the only, and best way, to do this? I would like to use a mixed model to account for some hierarchy in the data as well, but first need to sort out what family best describes my response variable.

Any help appreciated. Stata language especially appreciated!

Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ "I would like a "smoking gun" to say YES, YOU HAVE THE RIGHT FAMILY" - nothing will tell you this. The best you can hope for is a family that is not clearly wrong. There are many ways you can choose a distributional family, but in general it tends to involve a combination of a priori or theoretical considerations and the indications from the data itself. $\endgroup$
    – Glen_b
    Jul 9, 2013 at 2:30

2 Answers 2

Reset to default
14
$\begingroup$

I have some tips :

(1) How residuals ought to compare to fits isn't always all that obvious, so it's good to be familiar with diagnostics for particular models. In logistic regression models, for example, the Hosmer-Lemeshow statistic is used to assess goodness of fit; leverage values tend to be small where the estimated odds are very large, very small or about even; & so on.

(2) Sometimes one family of models can be seen as a special case of another, so you can use a hypothesis test on a parameter to help you choose. Exponential vs Weibull, for example.

(3) Akaike's Information Criterion is useful in choosing between different models, which includes choosing between different families.

(4) Theoretical/empirical knowledge about what you're modelling narrows the field of plausible models.

But there's no automatic way of finding the 'right' family; real-life data can come from distributions as complicated as you like, & the complexity of models that are worth trying to fit increases with the amount of data you have. This is part & parcel of Box's dictum that no models are true but some are useful.

Re @gung's comment: it appears the commonly used Hosmer-Lemeshow test is (a) surprisingly sensitive to the choice of bins, & (b) generally less powerful than some other tests against some relevant classes of alternative hypothesis. That doesn't detract from point (1): it's also good to be up-to-date.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks! Your suggestions are succinct and accurate. I am limited in the families I can use because of the structure of my response variable (positive, continuous, but highly skewed). Among the exponential family, it seems gamma is really the only option. In the meantime, I have found some useful tools by NJ Cox as appears in Stata Jounal 5(2): 259-273 - gammafit (estimates shape and scale parameters) and dpplot allows overlay of density probability plot and my response variable (can be done with many distributions and allows me to match the best family to my data).Thanks for other suggs too! $\endgroup$
    – RLang
    Nov 29, 2012 at 21:30
  • 1
    $\begingroup$ Note that the Hosmer-Lemeshow GoF test has been shown to depend on the binning used / be unreliable. $\endgroup$
    – gung - Reinstate Monica
    Dec 13, 2012 at 4:00
  • $\begingroup$ @Gung, It clearly depends on the binning used - not ideal, but not sure that's a big problem unless you start fiddling with the binnings to try for the result you want. How's it unreliable & what other tests are better? $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 13, 2012 at 9:31
  • 1
    $\begingroup$ See Frank Harrell's answer here: Stepwise model selection, Hosmer-Lemeshow statistics and prediction success of model in nested logistic regression in R for a discussion of these issues. $\endgroup$
    – gung - Reinstate Monica
    Dec 13, 2012 at 14:08
  • 1
    $\begingroup$ You're right that "invalid" is too strong; I only said "unreliable" & Harrell uses "obsolete", though. $\endgroup$
    – gung - Reinstate Monica
    Dec 13, 2012 at 15:09
8
$\begingroup$

You may find it interesting to read the vignette (introductory manual) for the R package fitdistrplus. I recognize that you prefer to work in Stata, but I think the vignette will be sufficiently self-explanatory that you can get some insights into the process of inferring distributional families from data. You will probably be able to implement some of the ideas in Stata via your own code. In particular, I think the Cullen and Frey graph, if it is / could be implemented in Stata, may be helpful for you.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I have revisited this problem again, and have switched to R and am using Zuur and Ieno as a guidance. Still many issues, but in general I think by using varIdent my model diagnostics look like they have 'minor heterogeneity'. Plotting residuals against fitted looks good, resids against each covariate provides some funky results for one of my model variables (elevation) - mostly a function of small sample size at high elevation. Thanks for your comment on fitdistrplus. Now that I'm using R and Rstudio (love it!) this will be handy! $\endgroup$
    – RLang
    Aug 14, 2013 at 22:20
  • 1
    $\begingroup$ The link is broken. Is this the intro manual you were talking about? cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf Or was it this one: cran.r-project.org/web/packages/fitdistrplus/vignettes/… $\endgroup$
    – emschorsch
    Feb 1, 2015 at 22:34
  • $\begingroup$ The latter link seems to be a different version of the vignette I was referring to. $\endgroup$
    – gung - Reinstate Monica
    Feb 1, 2015 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.