# Measure of absolute goodness-of-fit and good reference measure for cross validation in ordinal regression

I am following up the question I asked here with a more general one, as the reason why I am seeking to calculate the accuracy of my ordinal regression models is to use it as a measure of absolute goodness of fit as well as a reference measure in order to perform cross validation with my dataset.

In addition, I came across this discussion for the specific case of logistic regression, and the answer which seems to indicate that the examination of accuracy and deviance can give contradictory information.

1/ What would be a good reference measure to use for cross validation of an ordinal regression model? (For example with linear regression we would use the MSE but for GLMs it doesnt make sense?)

2/ If different from the above, what would be a good absolute goodness of fit measure - one which would be consistent with the relative goodness-of-fit I can determine by looking at deviance?

• Just a quick comment: MSE for logistic regression is perfectly sensible. It is often called Brier's score instead of MSE, but it is one of the proper scoring rules Frank Harrell refers to in the linked post. – cbeleites unhappy with SX Dec 6 '13 at 17:48
• It's good to know although the wikipedia article mentions that the Brier score is not applicable to ordinal models? – Neodyme Dec 9 '13 at 1:56
• The problem is that you don't know whether class 3 is twice as far from class 1 than class 2. – cbeleites unhappy with SX Dec 9 '13 at 11:31
• In general, and there are lots of specifics, I assert that the best measure of goodness of fit is one that the model optimises or a one-to-one function of it. Wanting to assess or report a model in terms of something different can make sense, but tread carefully. – Nick Cox Dec 9 '13 at 11:43
• I don't have much to add, except to give an example as you ask. I find corr(observed, expected) to be useful as a descriptive measure for generalized linear models even though in general that is not what is maximized in fitting, or even an equivalent. There is an argument at Zheng, B. and A. Agresti. 2000. Summarizing the predictive power of a generalized linear model. Statistics in Medicine 19: 1771-1781. The art with "figures of merit" is to use them for illumination, not support. – Nick Cox Dec 9 '13 at 15:03

(Disclaimer: I haven't had to do with ordinal regression so far, so these are just some ideas)

For what it's worth, I found this blog post:

http://fa.bianp.net/blog/2013/loss-functions-for-ordinal-regression/