0
$\begingroup$

Using glmnet, different metrics can be used to find the optimal value for log lambda, using cross-validation. For example, the maximum ROC-AUC for classification, or the minimum misclassification error rate.

Let's assume our glmnet model has a binary response (e.g., disease, yes vs. no).

Different steps in glmnet: (1) Define coefficient path for model predictors as function of log lambda. (2) X-fold cross-validation to find optimal log lambda that corresponds with lowest cross-validation misclassification error rate. (3) Apply log lambda that corresponds with lowest cross-validation misclassification error rate to find coefficients for markers in the glmnet model.

The misclassification error rate is a simple metric based on a confusion matrix. But, it requires a dichotomous variable (predicted disease, yes vs. no), not a continuous variable. So, in the glmnet algorithm, how exactely are the predicted classes defined?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Assuming glmnet restore a continuous variable: The decision where to cut who's positive and who's negative is yours to make, and in most times it's a "business" dilemma. If you, however, just interested in quantify how "good" is your model, you should measure it with AUC ROC. Here's an explanation on the matter: gim.unmc.edu/dxtests/roc3.htm $\endgroup$
    – Eran Moshe
    Commented Nov 27, 2017 at 14:24
  • 1
    $\begingroup$ "If you, however, just interested in quantify how "good" is your model, you should measure it with AUC ROC." I am going to have to disagree with this comment. Yes, AUC tell's you how good your model is at rank ordering, but it does not tell you about the accuracy of model, which has plenty of business application (reserves for banks comes to mind). I understand that CV has its limitations, but the way this comment is worded, it treats AUC as a one-size-fits-all metric, which it is not. $\endgroup$
    – Josh
    Commented Nov 27, 2017 at 16:07

1 Answer 1

Reset to default
1
$\begingroup$

I just ran across this question because I was wondering the same thing. I think I have it figured out, however. The misclassification error is a test/outsample error when using a 0/1 loss function, conditional on the used data set/model (with a given threshold). It averages out this error over all X and Y, even those not in the data set. In the spirit of cross validation, we are, however, almost always interested in the expected test error, which averages over the distribution of test data sets. The misclassification error you see as output of, for example, the plot of the cv.glmnet, is averaged out for every model you could employ with a varying threshold for every 'dataset' in the cross-validation procedure.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.