I understand that the ROC curve will plot the sensitivity vs FPR for varying thresholds.

For my SVM ML model, I desire a good sensitivity score so I have decreased the threshold to make a positive classification to 0.40. I know this threshold will increase the sensitivity but decrease the PPV and specificity.

The AUCROC for the default threshold SVM model is much higher (AUROC = 0.80) than the AUROC for the SVM model with the adjusted threshold of 0.40 (AUROC = 0.73).

This does not make sense to me because the ROC curve is a plot of the sensitivity vs FPR for varying thresholds. Shouldn't the ROC Curve and AUROC score be identical between the default svm model and the svm model with threshold = 0.40.

Or is it that I don't correctly understand what the ROC curve is plotting?

Here is the code I used:

prob_preds = clf.predict_proba(x_test)

threshold = 0.40
preds = [1 if prob_preds[i][1] > threshold else 0 for i in range(len(prob_preds))]

fpr, tpr, thresholds = roc_curve(y_test, preds)

  • $\begingroup$ If I understand correctly, you've created binary predictions and then constructed the ROC curve. This is not the same as using continuous-valued predictions and constructing a ROC curve. $\endgroup$ – Sycorax Aug 10 '20 at 16:38
  • $\begingroup$ Yes everything you wrote is correct. I have even added the code that I used to my question above. So if I constructed the ROC curve using continuous value predictions (like svc.decision_function(X_test)) how would that differ than constructing a ROC curve using binary predictions? $\endgroup$ – link Aug 10 '20 at 17:01
  • $\begingroup$ For fixed labels and predictions, predictions will have the same ROC curve. Same data in, same curve out. But thresholding the data changes the predictions, so the resulting ROC curve will depend on the choice of threshold. All ROC curves visualize the TPR/FPR tradeoff. But using the discrete data gives a less informative curve because there are only 2 predicted values. $\endgroup$ – Sycorax Aug 10 '20 at 17:09
  • $\begingroup$ Thank you Sycorax. Therefore, if I want to obtain high sensitivity scores, I can threshold the model to be more liberal in predicting the positive data class and only use those adjusted predictions to construct the confusion matrix. However when I construct ROC curves I should always use continuous valued predictions that are completely uninfluenced by thresholding (than the ROC curve will be the same all the time I assume). Is this all correct Sycorax? $\endgroup$ – link Aug 10 '20 at 17:17
  • 1
    $\begingroup$ @ping yes I understand that now. So in order to build ROC curves I should always use prediction probabilities. And if I want to obtain a higher sensitivity for my model, I can adjust the threshold, and use the adjusted threshold and adjusted predictions to make my confusion matrix? $\endgroup$ – link Aug 10 '20 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.