I am doing a penalized regression with categorical (ordinal) outcomes. I would like to select the shrinkage parameter $\lambda$ on the basis of cross-validation (CV).
In this case, I have 50k observations and my outcome variable has three (ordered) levels.
Three options occur to me:
- I normally see people using CV to pick a $\lambda$ by minimizing mis-classifcation error.
- I could also pick $\lambda$ in CV to maximize the joint log likelihood across folds.
- I could maximize the Spearman rank-correlation of the predicted class with the outcome.
Question: What are the considerations involved in picking among these cross-validation loss functions (a) in the ordinal case and (b) in the general multinomial case (for #1 and #2)?
Minimizing misclassification error seems most intuitive to me at first glance, but perhaps I'm missing something. Maybe something about bias/variance is going on here -- one answer here indicates that k-fold cross-validation is subject to substantial imprecision in general, but I don't know what the implications are for picking the cross-validation loss function.