1
$\begingroup$

I am doing a penalized regression with categorical (ordinal) outcomes. I would like to select the shrinkage parameter $\lambda$ on the basis of cross-validation (CV).

In this case, I have 50k observations and my outcome variable has three (ordered) levels.

Three options occur to me:

  1. I normally see people using CV to pick a $\lambda$ by minimizing mis-classifcation error.
  2. I could also pick $\lambda$ in CV to maximize the joint log likelihood across folds.
  3. I could maximize the Spearman rank-correlation of the predicted class with the outcome.

Question: What are the considerations involved in picking among these cross-validation loss functions (a) in the ordinal case and (b) in the general multinomial case (for #1 and #2)?

Minimizing misclassification error seems most intuitive to me at first glance, but perhaps I'm missing something. Maybe something about bias/variance is going on here -- one answer here indicates that k-fold cross-validation is subject to substantial imprecision in general, but I don't know what the implications are for picking the cross-validation loss function.

$\endgroup$
  • $\begingroup$ Maybe this question helps. In my experience, only deviance works well, as it is stated in the accepted answer. Hope it helps. $\endgroup$ – lrnzcig Apr 26 '16 at 9:11
1
$\begingroup$

I would avoid using the misclassification error, it is a discrete statistic and tends to be susceptible to over-fitting. I would opt for the log-likelihood, if it is a good statistic for fitting the model, it ought to be a good statistic for tuning the hyper-parameters. There is no real statistical distinction between parameters and hyper-parameters anyway, they are all parameters that must be estimated from the data anyway, so a consistent approach seems a reasonable thing to do.

Note also you may want to use the probabilities of class membership at some point (e.g. minimum risk classification, changing prior class probabilities etc.) so focusing on the decision boundary is not always a good idea.

$\endgroup$
  • $\begingroup$ Thanks for the answer - it's quite helpful. I didn't understand the second paragraph, though. What do you mean by focusing on the decision boundary? Is that a reference to some of the three options I enumerated in the question, namely #1 and #3? $\endgroup$ – Hatshepsut May 1 '16 at 0:54
  • $\begingroup$ The misclassification error rate is only sensitive to the value of the function near the decision boundary, whereas the likelihood depends on how well the model estimates class probablities everywhere. Whether this is important depends on whether you need well calibrated estimates of probability, or just a hard classification. $\endgroup$ – Dikran Marsupial May 5 '16 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.