I am running 9 regressions with 5 predictors each. The dependent variables are ordered, but have been treated before as count (and have a Gaussian distribution).

I ran the 9 regressions using the MASS package, using both 1) glm with family=gaussian, and 2) polr

The estimates of the models are very similar.

I want to determine pseudo $R^2$ now, and use the pR2function of the PSCL package. Again, $R^2$ values are basically identical for Gaussian and ordered models

However, the pseudo $R^2$ values are drastically higher than I had expected.

m1 0.13 0.27 0.32 0.42

m2 0.17 0.27 0.33 0.43

m3 0.06 0.19 0.36 0.40

m4 0.11 0.21 0.38 0.42

m5 0.14 0.26 0.39 0.45

m6 0.18 0.33 0.37 0.49

m7 0.10 0.28 0.29 0.41

m8 0.04 0.23 0.13 0.28

m9 0.07 0.31 0.12 0.36

The first column is $R^2$ obtained using linear regression models, the second McFadden pseudo $R^2$ , the third Maximum likelihood pseudo $r^2$, the fourth Cragg and Uhler's pseudo $r^2$, which is similar to Nagelkerke.

As far as I understand, Cragg and Uhler's pseudo $r^2$ has a range of 0-1, in contrast to the other pseudo $r^2$ values, and should be the one I should use. The values are between 3 and 7 times higher than the $r^2$ values obtained from linear regression models. Now I understand that linear regression doesn't fit very well because of the ordered nature of the variable, but a increase of 3-7 times sounds a bit too much.

Also my dependent variables are self reported single questionnaire items of a screening instrument for mental disorders, the predictors covariates like gender and personality traits. I am just skeptical that 40 or 50% variance can be explained by my covariates.

Maybe you have input on what might have gone wrong.

  • $\begingroup$ If the outcome variable is ordinal but not interval, the magnitude of the $R^2$ from the linear model is meaningless. If you change the labels in some non-linear (but still monotone) way, you can make the $R^2$ as large or as small as you want while still keeping the ordering in tact. $\endgroup$ – Macro Oct 22 '12 at 22:31
  • $\begingroup$ You could calculate the pseudo-$R^2$ for yourself, as a check. You could also look at the likelihood ratio test statistic against the linear model. As per Macro's advice, you don't need to dismiss them immediately. $\endgroup$ – gregmacfarlane Oct 23 '12 at 2:19

Remember that pseudo-$R^2$ does not have the same interpretation as does $R^2$ in a linear model, that is "$R^2$ percent of the variation in $Y$ is explained by variation in $X$." You can't say this about any of the pseudo-$R^2$ measures, which is why they are "pseudo."

This also why you shouldn't generally compare $R^2$ values to pseudo-$R^2$. If you must compare different models, you can calculate the log likelihood of the linear model and calculate its pseudo-$R^2$, or you can do a (non-nested) likelihood ratio test to see if the discrete option is more predictive than the simple linear model.

You can also use AIC to compare different classes of model.

  • $\begingroup$ So I can simply compare LLH ratios of ordered and linear models, taking into account their DF? polr() function im MASS for ordered models doesn't give me LLH values but "residual deviance", is that the same? Conversely, how would I calculate the LLH of the linear model and test it's pseudo-R^2? $\endgroup$ – Torvon Oct 23 '12 at 12:20
  • $\begingroup$ I don't think you can use the likelihood ratio test. At the very least some explanation of how the models are nested, which is a required conditioned for the LRT, is necessary. I'm having trouble seeing how they possibly could be. $\endgroup$ – Macro Oct 23 '12 at 12:47
  • $\begingroup$ You can calculate the likelihood at coefficients for a linear model in R using the logLik function. The null or constants likelihood can be calculated by estimating that model first. $\endgroup$ – gregmacfarlane Oct 23 '12 at 16:48
  • $\begingroup$ There exist non-nested likelihood ratio tests. dspace.cam.ac.uk/bitstream/1810/429/1/nnest.pdf $\endgroup$ – gregmacfarlane Oct 23 '12 at 16:51
  • $\begingroup$ @gmacfarlane, of course there are tests for non-nested models but the likelihood ratio test is a specific test, and it doesn't apply unless one model is a submodel of the other - see here for more info. If you have a specific test in mind (i.e. if you care to educate us on what a "non-nested likelihood ratio test" is) it might help if you gave some more details. I skimmed the link you posted and didn't see anything like that (of course, I don't have the 2-3 hours required to actually look carefully at it). $\endgroup$ – Macro Oct 23 '12 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.