in the most recent playground competition on kaggle (https://www.kaggle.com/c/tabular-playground-series-mar-2021/overview/evaluation) we once again have an evaluation via the area under the roc curve. When we use the .predict_proba methods (i.e. output probabilities instead of hard values 0, 1) we always get a significantly higher score.

But I do not quite understand how this is works for the ROC AUC metric. It makes sense for log loss as we actually calculate a difference, but does the True positive rate change? I think it shouldn't. Or is the metric actually doing some subtraction under that hood?

Thanks for any pointers!

  • $\begingroup$ How are you using log loss? // Do you used AUC as a loss function for training one model and log loss for another model? $\endgroup$ – Dave Mar 15 at 18:54
  • $\begingroup$ @Dave Hi, no, sorry, I was probably unclear. I mean, it makes sense if the metric were another, like log loss, that predicting probabilities could lead to a lower total loss. But it doesn't make sense for AUC to me. $\endgroup$ – Oliver Mar 15 at 18:55
  • $\begingroup$ To which score is AUC compared, accuracy? $\endgroup$ – Dave Mar 15 at 19:00
  • $\begingroup$ It is not compared, AUC is the score. Or maybe I am misunderstanding you? $\endgroup$ – Oliver Mar 15 at 19:05
  • $\begingroup$ You mention a "significantly higher score", so "higher" than what? $\endgroup$ – Dave Mar 15 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.