6
$\begingroup$

I have a binary classification task with classes 0 and 1 and the classes are unbalanced (class 1: ~8%). Data is in the range of ~10k samples and #features may vary but around 50-100.

I am only interested in the probability of an input to be in class 1 and I will use the predicted probability as an actual probability in another context later (see below).

Am am wondering how to best model this problem. My current approach isto use a random forest and predict_proba in scikit-learn and use ROC-AUC as a scoring function. The accuracy is 0.92 as it does not predict any class 1 with proba > 0.5.

After reading into the subject I came accross many suggestions and terms and I try to put a little structure in all of this. Specifically:

  1. I saw a couple of other scorers which were suggested, i.e. Cohens kapa, Matthews correlation coefficient, PC-AUC and some more. Should I look at all of those or is there a favorite for my problem?

  2. I just came accross the probability calibration subject in scikit. As I am interested in acual probability I think it's quite relevant. Am I right to assume that an additional CalibratedClassifierCV should be included in my model as it's based on Decision Trees? (is that done automatically in R?)

  3. After looking at some kaggle competitions xgboost seem very promising. Is that alsorithm well suited for my problem or do you have other suggestions regarding the algorithm (stick to the RF)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
10
$\begingroup$

You are considering different classifiers, but in fact this is not a classification problem. You are not interested in classifying your data as zeros and ones, but in predicting probabilities that individual cases are zeros and ones. In this case the usual method of choice, that is designed especially for such problems, is logistic regression. Contrary to popular beliefs, logistic regression is not a classifier, but rather it predicts probabilities, so it does exactly what you want.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes, naturally I tried the logistic regression first but it seems to work much worse than my current RF approach. I only did the grid search for a decent C value for it and it did not help much, thats why I switched to the RFin the first place. When I do 'predict_proba' in sklearn for LR, do I get actual probability values without using calibration? Also in the case of the RF, I thought, as it is a voting based system, that the probability given in sklearn would be the fraction of votes for class 1 and therefore does not need further calibration. Is that correct? $\endgroup$
    – jens0r
    Feb 22, 2017 at 15:05
  • $\begingroup$ @jens0r much worse in what sense..? How did you evaluate it? $\endgroup$
    – Tim ♦
    Feb 22, 2017 at 15:10
  • $\begingroup$ The main evaluation criteria I'm interested in is the output of a second model where I plug the probabilities in. But it would be good to have a direct, intermediate evaluation for the problem the question. For that I look at the AUC right now and I get better values for the RF model compared to the LR, roughly 0.77 to 0.79 (which is highly correlated to the evaluation measure in the subsequent model). $\endgroup$
    – jens0r
    Feb 22, 2017 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.