I'm trying to interpret examples from a probability perspective and my intuition is telling me Logistic Regression should be used for such a purpose despite the score being weaker than the other classifiers.

What I'm after is an accurate ranking order of photos. So if

  • photo1: 0.95
  • photo2: 0.6
  • photo3: 0.8

We could be fairly certain the ranking is photos 1,3,2... while this may seem obvious all the algos have different ranking orderings but I like the results from LR better but I don't want to be irrationally biased but he concept of distance from the average line seems appropriate in this case.

Which model should I use?


facial images projected to a smaller subspace using a CNN.

  • 2589 positive
  • 4911 negative

Using stratified 5-fold cross-val mean roc auc

  • Random Forest (example wtded): 87 (95 w basic tuning),
  • Logistic Regression(example wtded): 81
  • KNN: 95
  • XGB: 94

Inline with that, I don't understand the following quote from the XGBoost docs:

If you care only about the overall performance metric (AUC) of your prediction Balance the positive and negative weights via scale_pos_weight Use AUC for evaluation

If you care about predicting the right probability In such a case, you cannot re-balance the dataset Set parameter max_delta_step to a finite number (say 1) to help convergence

I thought ROC AUC was derived from the probabilities of the examples. Does this mean the individual examples probabilities aren't as valid in a balanced model (cost sensitive learning)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.