0
$\begingroup$

I built different models to approach the anomaly detection problem and I'd like to plot a ROC curve to see how do they perform on my datasets. Both the models are unsupervised neural networks and they give me as output the predicted labels for each point (it can be = 0 if it is not an anomaly, 1 if it is). I have the correct labels and I can manually calculate parameters such as precision, recall etc.

The problem is that when I give these parameters to the roc plot dedicated method, it obviously doesn't work since it requires a probability. How can I solve this problem? Are there valid alternatives to the ROC curve? I'll leave my (wrong) code so you can check what I tried to do:

import matplotlib.pyplot as plt
import sklearn.metrics as metrics

y_test = [1, 1, 1, 1, 1, 1, 1, 1]
preds = [1, 1, 1, 1, 1, 1, 1, 1]
fpr, tpr, threshold = metrics.roc_curve(y_test, preds)
roc_auc = metrics.auc(fpr, tpr)
plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

When I run it I get an empty plot and the following warning:

/opt/anaconda3/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:941: UndefinedMetricWarning: No negative samples in y_true, false positive value should be meaningless warnings.warn("No negative samples in y_true, "

$\endgroup$
3
  • 3
    $\begingroup$ What makes the points on the ROC curve useful in this setting? Why do you think precision and recall are useful? fharrell.com/post/mlconfusion $\endgroup$ Oct 10, 2021 at 17:57
  • $\begingroup$ Why don’t you have the probability values? $\endgroup$
    – Dave
    Oct 10, 2021 at 21:02
  • $\begingroup$ "It obviously doesn't work since it requires a probability": what makes you think so? $\endgroup$
    – Calimo
    Oct 11, 2021 at 6:07

1 Answer 1

2
$\begingroup$

Building the ROC curve with sklearn.metrics requires to provide the score and not the final classification: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html Then the decision threshold on the score is varied and the resulting FPR and TPR as a parametric function of the threshold is reported.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.