3
$\begingroup$

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether you have a good model?

Please provide me with any reference material that I could read more on this topic. Websites, articles or book references on this topic will be a great help. I am kind of stuck!

I have R, SAS, Stata, SPSS, and Minitab available.

$\endgroup$
  • $\begingroup$ Andy W Dr Wolfgang Langer. The assessment of fit in the class of logistic regression models: A pathway out of the jungle of Pseudo R's. Just google that document, you will get the information and citations there $\endgroup$ – user51984 Jul 13 '14 at 1:43
3
$\begingroup$

I would consider these tests at a minimum:

  • Hausman or Small-Hsiao tests of the IIA
  • Confusion matrices of predicted vs. actual outcome
  • Information criteria (AIC, BIC)
  • Various scalar measures of fit (like McFadden's $R^2$)
  • Wald or LR tests for combining alternatives

With Stata, check out the SPost from Long and Freese as their as their categorical variables book for code and a nice intro to all of these tests with examples.

$\endgroup$
  • $\begingroup$ Using confusion matrices is akin to using % predicted, which is an improper scoring rule and could potentially lead to favoring a 'bad' model.. A proper scoring rule as well as McFadden's $R^2$ and other likelihood measures will reward models which better capture the true probabilities. $\endgroup$ – user44764 Jun 10 '14 at 1:24
  • $\begingroup$ @Matthew I would not advocate using CMs (or any univariate measures of fit) for automated model selection. I don't think the OP was asking about model selection. But as a measure of fit that tells you if a given model puts people in the right buckets, or perhaps on a hold-out sample, CMs can be useful. $\endgroup$ – Dimitriy V. Masterov Jun 10 '14 at 1:37
  • $\begingroup$ It seems strange to me to be in a position where I would be evaluating the fit of my model without comparing it to other possible fits, but after a careful reread of the question I think you're right about what is being asked for. $\endgroup$ – user44764 Jun 10 '14 at 1:48
2
$\begingroup$

I like going through the tutorials on IDRE. Here are the R ones, they have them for SAS as well:

http://www.ats.ucla.edu/stat/r/dae/logit.htm

http://www.ats.ucla.edu/stat/r/dae/mlogit.htm

http://www.ats.ucla.edu/stat/r/dae/probit.htm

Here is what the multinomial article says about diagnostics:

Diagnostics and model fit: Unlike logistic regression where there are many statistics for performing model diagnostics, it is not as straightforward to do diagnostics with multinomial logistic regression models. For the purpose of detecting outliers or influential data points, one can run separate logit models and use the diagnostics tools on each model.

From the logistic article: Diagnostics: The diagnostics for logistic regression are different from those for OLS regression. For a discussion of model diagnostics for logistic regression, see Hosmer and Lemeshow (2000, Chapter 5). Note that diagnostics done for logistic regression are similar to those done for probit regression.

$\endgroup$
  • $\begingroup$ Great reference, I think I will find there almost everything I need right now plus a lot more reference material. Thanks a lot! $\endgroup$ – rbetan Jan 30 '14 at 22:18
  • $\begingroup$ @rebetan, if this answer was appropriate feel free to mark is as correct. $\endgroup$ – bdeonovic May 8 '14 at 19:38
0
$\begingroup$

according to Mc Fadden the rule of thumb is that pseudo r2 should be between .20 and .40 for mnl. Am also having problems with the same, you can also use pearson and deviance if using SPSS, the chi2 value should not be significant.

$\endgroup$
  • 1
    $\begingroup$ Do you care to provide a citation for the rule of thumb? $\endgroup$ – Andy W Jun 27 '14 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.