I am trying to justify to the use of an ordered probit, my dependent variable is a survey response on a likert scale so is likely ordinal, but I wanted to provide a goodness of fit stat to back up my claim that the ordered probit is better?

Would AIC or BIC be appropriate or are they only for validating extra variables? Otherwise I was thinking of using a pseudo $R^2$ or log-likelihood value. (In my case the loglikelihood, AIC and a BIC are lower for the oprobit, whilst for the pseudo R squared, only the McFadden R squared is higher for the oprobit whilst the rest are lower for OLS.)

  • 1
    $\begingroup$ This seems backward to me, but only you know your intended readership. Given a Likert scale for a response, I would expect all the scepticism to be directed at linear regression (why people mention the estimation method, OLS, rather than the model is a puzzle) as at best a dubious exercise deserving justification. In any case, the comparison makes limited sense as linear regression is here trying to predict the graded response while ordinal models are focused rather on their probabilities. Similarly, the likelihood functions aren't comparable. $\endgroup$ – Nick Cox Oct 23 '18 at 14:11
  • $\begingroup$ I don't understand what you mean by backwards? I stated in my methodology that I will use ordered probit over OLS due to the assumption that my dependent variable is ordinal. I writing the regression diagnostics and I was just looking at justifying this assumption with some form of goodness of fit stat to justify my specification of the OProbit over OLS with a formal test, if that makes things any clearer? $\endgroup$ – Martin Oct 23 '18 at 16:20
  • $\begingroup$ I meant this: Your emphasis is on justifying the better method. If anything needs defence, it is linear regression which treats the scale points as equally spaced and (unless you are doing something else which you aren't telling us) fits lines or surfaces that won't respect the bounds of the scale. I don't think there is a formal test here because there is no framework within which these methods are special cases. It's like comparing hockey and football. Also, goodness of fit comparisons depend on trying to predict the same thing, which the models are not doing. $\endgroup$ – Nick Cox Oct 23 '18 at 16:34
  • $\begingroup$ Cross-posted at statalist.org/forums/forum/general-stata-discussion/general/… It's a good idea to tell people about cross-posting, even if you didn't get a quick answer in the other place. $\endgroup$ – Nick Cox Oct 23 '18 at 17:08
  • $\begingroup$ I see what you mean thanks! I was just confused as I read in a Stata textbook that I should use fitstat as a regression diagnostic, however I didn't know what to compare my ordered probit to. Just as a side question, I was wondering aside from checking for outliers, the parallel lines assumption and multicollinearity do you now if there are any other tests to do on my model in the diagnostics? Any advice would be highly appreciated. $\endgroup$ – Martin Oct 23 '18 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.