0
$\begingroup$

I'm trying to interpret examples from a probability perspective and my intuition is telling me Logistic Regression should be used for such a purpose despite the score being weaker than the other classifiers.

What I'm after is an accurate ranking order of photos. So if

  • photo1: 0.95
  • photo2: 0.6
  • photo3: 0.8

We could be fairly certain the ranking is photos 1,3,2... while this may seem obvious all the algos have different ranking orderings but I like the results from LR better but I don't want to be irrationally biased but he concept of distance from the average line seems appropriate in this case.

Which model should I use?

data:

facial images projected to a smaller subspace using a CNN.

  • 2589 positive
  • 4911 negative

Using stratified 5-fold cross-val mean roc auc

  • Random Forest (example wtded): 87 (95 w basic tuning),
  • Logistic Regression(example wtded): 81
  • KNN: 95
  • XGB: 94

Inline with that, I don't understand the following quote from the XGBoost docs:

If you care only about the overall performance metric (AUC) of your prediction Balance the positive and negative weights via scale_pos_weight Use AUC for evaluation

If you care about predicting the right probability In such a case, you cannot re-balance the dataset Set parameter max_delta_step to a finite number (say 1) to help convergence

I thought ROC AUC was derived from the probabilities of the examples. Does this mean the individual examples probabilities aren't as valid in a balanced model (cost sensitive learning)?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.