1
$\begingroup$

Task

Binary classification for an imbalanced dataset with 70 positive samples and 700 negative samples

Data

Oversample positive samples to match the number of negatives. Each sample has 10 features.

Model

By cross-validation, the tree-based methods gave better performace than Logistic regression and two-layed dense neural networks. The methods in use are the XGBoost api for Sklearn and the RandomForestClassifier in sklearn.

Training result

Normalized confusiont matrix for testing data by XGBoost: $$\quad\quad true\quad false$$ $$true\quad 0.71\quad 0.29$$ $$false\quad 0.33\quad 0.67$$

Normalized confusiont matrix for testing data by Random Forest: $$\quad\quad true\quad false$$ $$true\quad 0.77\quad 0.23$$ $$false\quad 0.27\quad 0.73$$

The hyperparameters settings

PARAM_CONFIG = { 'random_forest':{ 'n_estimators':80, 'criterion':'entropy', 'min_samples_leaf':20, 'max_depth':3 }, 'xgboost':{ 'max_depth':3, 'learning_rate':0.0001, 'objective':'binary:logistic', 'scale_pos_weight': 1, 'n_estimators': 50, 'subsample': 0.8 } }

Predicted probability on independent dataset

enter image description here

The question

Now comes to my problem, the model performances from training are very close for both methods. But when I looked into the predicted probabilities, XGBoost gives always marginal probabilities, which is not case for random forest.

XGBoost and Random Forest gave the same prediction accuracy but XGBoost are being less confident for all samples. What could go wrong here? or How should one explain this?

$\endgroup$
  • 1
    $\begingroup$ This clustering of the probabilities appears to be a known "feature" of gradient boosting: see datascience.stackexchange.com/questions/14527/… for a different version of the same problem. I wouldn't take the XGBoost outputs as real predicted probabilities, myself. $\endgroup$ – jbowman Feb 1 '18 at 20:45
  • $\begingroup$ @jbowman Thanks a lot for the link. I will do more test to find out why the xgboost scores are slightly above 0.5. The probability calibration might alter the clustering a bit. $\endgroup$ – doubllle Feb 2 '18 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.