2
$\begingroup$

I am working on a binary classification task on imbalanced data.

Since the accuracy is not so meaningful in this case. I use Scikit-Learn to compute the Precision-Recall curve and ROC curve in order to evaluate the model performance.

But I found both of the curves would be a horizontal line when I use Random Forest with a lot of estimators, it also happens when I use a SGD classifier to fit it.

The ROC chart is as following:

enter image description here

And the Precision-Recall chart:

enter image description here

Since Random Forest behaves randomly, I don't get a horizontal line in every run, sometimes I also get a regular ROC and PR curve. But the horizontal line is much more common.

Is this normal? Or I made some mistakes in my code?

Here is the snippet of my code:

classifier.fit(X_train, Y_train)
try:
    scores = classifier.decision_function(X_test)
except:
    scores = classifier.predict_proba(X_test)[:,1]

precision, recall, _ = precision_recall_curve(Y_test, scores, pos_label=1)
average_precision = average_precision_score(Y_test, scores)

plt.plot(recall, precision, label='area = %0.2f' % average_precision, color="green")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision Recall Curve')
plt.legend(loc="lower right")
plt.show()
$\endgroup$
  • $\begingroup$ This is normal. $\endgroup$ – Marc Claesen Jul 14 '15 at 6:25
  • $\begingroup$ Why are you using 1.05 in ylim and the exact bound otherwise? $\endgroup$ – Calimo Jul 14 '15 at 6:41
  • $\begingroup$ Calimo: I copy this code from the scikit-learn example, The original setting is like that $\endgroup$ – Jim GB Jul 14 '15 at 10:37
  • $\begingroup$ Then you may want to change it and answer your question yourself $\endgroup$ – Calimo Jul 14 '15 at 14:53
0
$\begingroup$

I got the answer from my duplicate post here:

Is it possible that Precision-Recall curve or a ROC curve is a horizontal line?

The horizontal lines are possible but not normal. The reason I got horizontal is that I happen to choose a very easy testing data.

To solve this. Just simply apply the Stratified Cross Validation to get a more generalized calculation.

The following charts are what I got:

enter image description here

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.