0
$\begingroup$

tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area under the ROC curve?

~~~~~

More details

I am using the glmnet package in R to perform elastic-net penalized logistic regression for binary classification on a severely class unbalanced dataset, using type.measure = 'auc' to optimize the area under the curve (AUC) of the receiver operator characteristic (ROC), during cross-validation to select an elastic-net lambda parameter.

However, with severely imbalanced datasets, it appears that area under the Precision-Recall (PRC) curve may be preferable to ROC AUC; e.g., Saito 2015.

This does not seem to be a type.measure option in cv.glmnet. Has anyone found a way to use glmnet logistic regression with PRC-AUC? If not, how important do people think it is to use PRC and not ROC for a severely class imbalanced target?

$\endgroup$
  • $\begingroup$ Questions that are only about software (e.g. error messages, code or packages, etc.) are generally off topic here. Only your last question is on topic. $\endgroup$ – gung May 22 at 3:53
2
$\begingroup$

The gold standard objective function is the log-likelihood or penalized log-likelihood. Statistical models are not built to optimize anything other than that. Also you need to take time to understand proper accuracy scoring rules. Once you are done with fitting the model you can compute the concordance probability ($c$-index; AUROC) to quantify pure predictive discrimination.

Note that elastic net has two penalty parameters, and it is worthwhile to bootstrap the whole process to see whether the set of features selected is reproducible. It may not be. For that reason, ridge regression offers better predictive performance.

$\endgroup$
  • $\begingroup$ Correct me if I'm wrong, but in the OP's case they are not optimizing against ROC, rather just cross-validating the choice of hyper-parameters against ROC. The objective function does not change throughout this exercise. $\endgroup$ – Nutle May 22 at 9:04
  • 1
    $\begingroup$ Yes I understood that. Rank concordance measures such as AUROC are not sensitive enough for this purpose. The log-likelihood (deviance) needs to be used. AUROC is not a proper accuracy score, i.e., it can be fooled by the wrong model or a miscalibrated model. $\endgroup$ – Frank Harrell May 22 at 11:01
1
$\begingroup$

It is important to use the Precision Recall AUC as opposed to ROC AUC for imbalanced datasets. I use glmnet with my logistic regression models, but I do my own cross validation and select the minimum lambda with the maximum PR AUC. Make sure to stratify the folds to ensure you have at least one member of the smaller class in each fold. Weight your classes according to the imbalance ratio. It also helps to create multiple models assuming your cross validation folds are different each time and then average the lambda across all your models to get a final lambda.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.