1
$\begingroup$

I am doing classification on a fairly imbalanced dataset (about 1:2 ratio). I have so far so far tried lasso and logistic regression. I didn't downsample the dataset because the sample size is low (about 1,300). You can find the PR curve here and the ROC curve here. As you can see, they look weird. I was under the impression that by increasing/decreasing the threshold, the measures Precision and recall go in the opposite direction which is clearly not the case here. AUC value is about 0.43 for lasso and 0.54 for logistic regression. For some thresholds, I am getting decent Recall and Specificity. Would downsampling help?

I am not sure if this is correct or if these models are even useful. Any feedback/suggestion is greatly appreciated.

Thanks!

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

Neither precision nor recall considers the cases with true negative classifications. That's also true of F-scores, which are based on precision and recall.

In contrast, the ROC curve is based on recall (usually called sensitivity in that context) and specificity (the true-negative rate).

So, putting aside any possible technical errors, my guess is that the apparent discrepancy has to do with your (lack of) success at identifying true negatives.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I agree. The models are poor at identifying true negatives. I have never come across non-concave roc curves. So was wondering if it was even technically possible. $\endgroup$
    – Mohamad Sahil
    Commented Jan 27, 2021 at 21:06
  • $\begingroup$ @MohamadSahil although the ROC curves are non-concave, they still are non-decreasing so they are valid. The horizontal segments that give non-concavity just represent large increases in the false-positive rate, FP/(FP+TN), with no corresponding improvement in recall/sensitivity, for certain changes in the probability cutoff as you move along the ROC curve. Each of those horizontal segments might correspond to individual or just a handful of cases. They have little to do with the particulars of your question, although ROC applications are better done with a concave hull, as noted. $\endgroup$
    – EdM
    Commented Jan 27, 2021 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.