2
$\begingroup$

I'm practicing my machinelearning / sklearn skills with a kaggle playground and I'm having trouble why a suggested change made by a fellow user yields better results. The challenge involves binary classification and the results are ranked by their roc_auc score.

I created a classifier (clf) using LogisticRegression and made the predictions with the following line of code

predictions = clf.predict(X)

This method resulted in a score of 0.68.

A fellow user suggested that I instead used the following method:

predictions = clf.predict_proba(X)[:,1]

This did indeed result in a higher score (0.76), but I'm having trouble understanding why.

As I understand it, the predict method takes the highest scoring class of the predict_proba as its output and thus the score between both examples should be the same.

e.g:

prob_0 | prob_1 | predict

 0.3      0.7       1
 0.51     0.49      0

The higher score of the predict_proba method indicates that my understanding of these methods is flawed, but I'm not sure what I'm getting wrong here. Could anyone help point me in the right direction?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Your understanding of the method is correct. However, I find it weird that you can simply submit your probabilities [0,1] rather than actual class prediction {0,1}. Maybe it computes the AUC automatically within Kaggle for several threshold and reports the best one? Can you link the Kaggle discussion please. $\endgroup$
    – Tom
    Commented Feb 10, 2019 at 0:39
  • 1
    $\begingroup$ Maybe you want to read how to calculate auc. ( One view is basically sweeping decision threshold from zero to one). If you only provide preclassified values (ie zero and 1), then how do you do this... $\endgroup$
    – seanv507
    Commented Feb 10, 2019 at 0:59
  • $\begingroup$ Thanks for your responses! I received the tip in my kernel (kaggle.com/jasperkoops/logistic-model-with-pipeline), I've included the link in the original post as well. (It probably should have been there from the beginning) $\endgroup$
    – Jasper
    Commented Feb 10, 2019 at 1:16

1 Answer 1

Reset to default
1
$\begingroup$

The Receiver Operating Characteristic (ROC) Curve is computed by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) at uniformly distributed threshold values from 0 to 1. The Area Under the Curve (AUC) is then calculated to turn this into a numerical score.

This means that you need to provide a vector of probabilities in order for the various thresholds to be meaningful. SKlearn's predict function simply provides binary value based on a threshold of 0.5. This will be interpreted as a vector of probabilities that are either 0% or 100%, making any changes in the threshold value irrelevant.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.