1
$\begingroup$

tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area under the ROC curve?

~~~~~

More details

I am using the glmnet package in R to perform elastic-net penalized logistic regression for binary classification on a severely class unbalanced dataset, using type.measure = 'auc' to optimize the area under the curve (AUC) of the receiver operator characteristic (ROC), during cross-validation to select an elastic-net lambda parameter.

However, with severely imbalanced datasets, it appears that area under the Precision-Recall (PRC) curve may be preferable to ROC AUC; e.g., Saito 2015.

This does not seem to be a type.measure option in cv.glmnet. Has anyone found a way to use glmnet logistic regression with PRC-AUC? If not, how important do people think it is to use PRC and not ROC for a severely class imbalanced target?

Improve this question
$\endgroup$
1
  • $\begingroup$ Questions that are only about software (e.g. error messages, code or packages, etc.) are generally off topic here. Only your last question is on topic. $\endgroup$
    – gung - Reinstate Monica
    Commented May 22, 2019 at 3:53

2 Answers 2

Reset to default
1
$\begingroup$

The gold standard objective function is the log-likelihood or penalized log-likelihood. Statistical models are not built to optimize anything other than that. Also you need to take time to understand proper accuracy scoring rules. Once you are done with fitting the model you can compute the concordance probability ($c$-index; AUROC) to quantify pure predictive discrimination.

Note that elastic net has two penalty parameters, and it is worthwhile to bootstrap the whole process to see whether the set of features selected is reproducible. It may not be. For that reason, ridge regression offers better predictive performance.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Correct me if I'm wrong, but in the OP's case they are not optimizing against ROC, rather just cross-validating the choice of hyper-parameters against ROC. The objective function does not change throughout this exercise. $\endgroup$
    – runr
    Commented May 22, 2019 at 9:04
  • 1
    $\begingroup$ Yes I understood that. Rank concordance measures such as AUROC are not sensitive enough for this purpose. The log-likelihood (deviance) needs to be used. AUROC is not a proper accuracy score, i.e., it can be fooled by the wrong model or a miscalibrated model. $\endgroup$
    – Frank Harrell
    Commented May 22, 2019 at 11:01
  • $\begingroup$ In your answer, the first point is not a real answer to the question. And the elastic net is a generalization of ridge regression (ridge regression is a particular case of the elastic net with a penalized parameter set to zero), so I don't understand your second point. $\endgroup$
    – mrb
    Commented Nov 24, 2019 at 14:08
  • $\begingroup$ Elastic net is penalized maximum likelihood estimation as you stated. To fit it you optimize the penalized log likelihood function. AUROC plays no role. $\endgroup$
    – Frank Harrell
    Commented Nov 24, 2019 at 15:42
1
$\begingroup$

It is important to use the Precision Recall AUC as opposed to ROC AUC for imbalanced datasets. I use glmnet with my logistic regression models, but I do my own cross validation and select the minimum lambda with the maximum PR AUC. Make sure to stratify the folds to ensure you have at least one member of the smaller class in each fold. Weight your classes according to the imbalance ratio. It also helps to create multiple models assuming your cross validation folds are different each time and then average the lambda across all your models to get a final lambda.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.