Compare fit of ordinal to linear model I´m doing multivariate modeling on a dataset where the dependent variable is ordinal-values running from 0-11. So far I´ve done general linear models and it runs alright. However, I´d also like to run it as an logistic ordinal regression. 


*

*What is the recommended maximum number of categories in the dependent ordinal variable?    

*Would I need to cut the variable into groups and reduce the number of possible categories?

*How can I compare the fit of an ordinal-outcome model to a linear regression model? Especially, if the ordinal variable has been cut into fewer groups?

 A: It is a great idea to compare the two methods, because it often teaches us that the semiparametric proportional odds ordinal logistic model is a better fit than OLS.  There are at least three direct ways to check this.


*

*Compute the mean absolute difference between predicted and observed $Y$ for both methods.  For the P.O. model you can estimate the mean or median $Y|X$.

*Compute Kendall's $\tau$ between predicted and observed $Y$, separately for the two methods to measure predictive discrimination using a rank correlation measure.

*If there were only one $X$ and it was categorical, you can look at the distribution of $Y$ stratified by $X$.  OLS assumes [if you are computing $P$-values or confidence limits] that the normal inverse transformation of the empirical CDF of $Y$ when stratified by $X$ consists of parallel straight lines.  The P.O. model assumes that the logit of the CDF consists of parallel (but not necessarily straight) lines.

A: *

*You can do ordinal logistic with 11 categories, as long as there is sufficient N in each category. The rule of thumb is that the category with the smallest N should have 10 per independent variable.

*As far as I know, you can't directly compare the fits with some statistic that is output from both runs; what I often do is to make plots of the predicted values from each model: A scatter plot of one vs. the other, a box plot of the differences between them and sometimes a mean difference plot (this may be less familiar; it is a creation of John Tukey's. Plot the mean of the two on the x axis and the difference on the y axis)
Note: You do not need more than one observation per level of $Y$ if the ordering of $Y$ is used in the analysis (which is the case for the usual proportional odds model).  You only need multiple observations if you are treating $Y$ categories as nominal (polytomous).  The proportional odds model generalizes the Wilcoxon and Spearman tests which do not require ties in $Y$.
The only time you need to collapse $Y$ categories in ordinal logistic regression is when you are assessing the proportional odds assumption.
