Selecting a loss-function for k-fold cross-validation over shrinkage parameter

Question

I am doing a penalized regression with categorical (ordinal) outcomes. I would like to select the shrinkage parameter $\lambda$ on the basis of cross-validation (CV).

In this case, I have 50k observations and my outcome variable has three (ordered) levels.

Three options occur to me:

1. I normally see people using CV to pick a $\lambda$ by minimizing mis-classifcation error.
2. I could also pick $\lambda$ in CV to maximize the joint log likelihood across folds.
3. I could maximize the Spearman rank-correlation of the predicted class with the outcome.

Question: What are the considerations involved in picking among these cross-validation loss functions (a) in the ordinal case and (b) in the general multinomial case (for #1 and #2)?

Minimizing misclassification error seems most intuitive to me at first glance, but perhaps I'm missing something. Maybe something about bias/variance is going on here -- one answer here indicates that k-fold cross-validation is subject to substantial imprecision in general, but I don't know what the implications are for picking the cross-validation loss function.

Answer 1

I would avoid using the misclassification error, it is a discrete statistic and tends to be susceptible to over-fitting. I would opt for the log-likelihood, if it is a good statistic for fitting the model, it ought to be a good statistic for tuning the hyper-parameters. There is no real statistical distinction between parameters and hyper-parameters anyway, they are all parameters that must be estimated from the data anyway, so a consistent approach seems a reasonable thing to do.

Note also you may want to use the probabilities of class membership at some point (e.g. minimum risk classification, changing prior class probabilities etc.) so focusing on the decision boundary is not always a good idea.