3
$\begingroup$

I am following up the question I asked here with a more general one, as the reason why I am seeking to calculate the accuracy of my ordinal regression models is to use it as a measure of absolute goodness of fit as well as a reference measure in order to perform cross validation with my dataset.

In addition, I came across this discussion for the specific case of logistic regression, and the answer which seems to indicate that the examination of accuracy and deviance can give contradictory information.

1/ What would be a good reference measure to use for cross validation of an ordinal regression model? (For example with linear regression we would use the MSE but for GLMs it doesnt make sense?)

2/ If different from the above, what would be a good absolute goodness of fit measure - one which would be consistent with the relative goodness-of-fit I can determine by looking at deviance?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Just a quick comment: MSE for logistic regression is perfectly sensible. It is often called Brier's score instead of MSE, but it is one of the proper scoring rules Frank Harrell refers to in the linked post. $\endgroup$
    – cbeleites
    Commented Dec 6, 2013 at 17:48
  • $\begingroup$ It's good to know although the wikipedia article mentions that the Brier score is not applicable to ordinal models? $\endgroup$
    – Neodyme
    Commented Dec 9, 2013 at 1:56
  • $\begingroup$ The problem is that you don't know whether class 3 is twice as far from class 1 than class 2. $\endgroup$
    – cbeleites
    Commented Dec 9, 2013 at 11:31
  • $\begingroup$ In general, and there are lots of specifics, I assert that the best measure of goodness of fit is one that the model optimises or a one-to-one function of it. Wanting to assess or report a model in terms of something different can make sense, but tread carefully. $\endgroup$
    – Nick Cox
    Commented Dec 9, 2013 at 11:43
  • 2
    $\begingroup$ I don't have much to add, except to give an example as you ask. I find corr(observed, expected) to be useful as a descriptive measure for generalized linear models even though in general that is not what is maximized in fitting, or even an equivalent. There is an argument at Zheng, B. and A. Agresti. 2000. Summarizing the predictive power of a generalized linear model. Statistics in Medicine 19: 1771-1781. The art with "figures of merit" is to use them for illumination, not support. $\endgroup$
    – Nick Cox
    Commented Dec 9, 2013 at 15:03

1 Answer 1

Reset to default
2
$\begingroup$

(Disclaimer: I haven't had to do with ordinal regression so far, so these are just some ideas)

For what it's worth, I found this blog post:

http://fa.bianp.net/blog/2013/loss-functions-for-ordinal-regression/

linking to

JDM Rennie, N Srebro: Loss functions for preference levels: Regression with discrete ordered labels, Proceedings of the IJCAI Multidisciplinary Workshop on Advances in Preference Handling 2005

Note that their added-up hinge loss would go to MSE if the thresholds were equidistant and infinitely close. The problem is of course the equidistant.

I think I'd try two things: - What happens with MSE (or similar) nevertheless (i.e. assuming equidistance) - MSE can be used if the classes are treated as independent (i.e. not ordered)
With that approach, you say that a deviation of 2 is just as bad as it is, regardless in which class the prediction ends up in the end.

I'd have a look what happens with these two approaches. Maybe that gives an idea how to go on.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, this is an interesting paper. I will try both the MSE and one of these loss functions and see what I get. $\endgroup$
    – Neodyme
    Commented Dec 10, 2013 at 4:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.