5
$\begingroup$

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether you have a good model?

Please provide me with any reference material that I could read more on this topic. Websites, articles or book references on this topic will be a great help. I am kind of stuck!

I have R, SAS, Stata, SPSS, and Minitab available.

Improve this question
$\endgroup$
2
  • $\begingroup$ according to Mc Fadden the rule of thumb is that pseudo r2 should be between .20 and .40 for mnl. Am also having problems with the same, you can also use pearson and deviance if using SPSS, the chi2 value should not be significant. $\endgroup$
    – Davis
    Commented Jun 27, 2014 at 10:08
  • $\begingroup$ Andy W Dr Wolfgang Langer. The assessment of fit in the class of logistic regression models: A pathway out of the jungle of Pseudo R's. Just google that document, you will get the information and citations there $\endgroup$
    – user51984
    Commented Jul 13, 2014 at 1:43

2 Answers 2

Reset to default
4
$\begingroup$

I would consider these tests at a minimum:

  • Hausman or Small-Hsiao tests of the IIA
  • Confusion matrices of predicted vs. actual outcome
  • Information criteria (AIC, BIC)
  • Various scalar measures of fit (like McFadden's $R^2$)
  • Wald or LR tests for combining alternatives

With Stata, check out the SPost from Long and Freese as their as their categorical variables book for code and a nice intro to all of these tests with examples.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Using confusion matrices is akin to using % predicted, which is an improper scoring rule and could potentially lead to favoring a 'bad' model.. A proper scoring rule as well as McFadden's $R^2$ and other likelihood measures will reward models which better capture the true probabilities. $\endgroup$
    – user44764
    Commented Jun 10, 2014 at 1:24
  • $\begingroup$ @Matthew I would not advocate using CMs (or any univariate measures of fit) for automated model selection. I don't think the OP was asking about model selection. But as a measure of fit that tells you if a given model puts people in the right buckets, or perhaps on a hold-out sample, CMs can be useful. $\endgroup$
    – dimitriy
    Commented Jun 10, 2014 at 1:37
  • $\begingroup$ It seems strange to me to be in a position where I would be evaluating the fit of my model without comparing it to other possible fits, but after a careful reread of the question I think you're right about what is being asked for. $\endgroup$
    – user44764
    Commented Jun 10, 2014 at 1:48
3
$\begingroup$

I like going through the tutorials on IDRE. Here are the R ones, they have them for SAS as well:

http://www.ats.ucla.edu/stat/r/dae/logit.htm

http://www.ats.ucla.edu/stat/r/dae/mlogit.htm

http://www.ats.ucla.edu/stat/r/dae/probit.htm

Here is what the multinomial article says about diagnostics:

Diagnostics and model fit: Unlike logistic regression where there are many statistics for performing model diagnostics, it is not as straightforward to do diagnostics with multinomial logistic regression models. For the purpose of detecting outliers or influential data points, one can run separate logit models and use the diagnostics tools on each model.

From the logistic article:

Diagnostics: The diagnostics for logistic regression are different from those for OLS regression. For a discussion of model diagnostics for logistic regression, see Hosmer and Lemeshow (2000, Chapter 5). Note that diagnostics done for logistic regression are similar to those done for probit regression.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Great reference, I think I will find there almost everything I need right now plus a lot more reference material. Thanks a lot! $\endgroup$
    – rbetan
    Commented Jan 30, 2014 at 22:18
  • $\begingroup$ @rebetan, if this answer was appropriate feel free to mark is as correct. $\endgroup$
    – bdeonovic
    Commented May 8, 2014 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.