1
$\begingroup$

Look at the following sample code to understand the problem. I am facing this problem in a real dataset but to represent the issue I am creating a random dataframe below.

random_df <- data.frame(replicate(10,sample(0:1,100000,rep=TRUE)))
pred <- sample(0:1,100000,prob=c(0.99,0.01),rep=T)
random_df[] <- lapply(random_df, as.numeric)


basic_train <-xgboost(data=data.matrix(random_df),label=pred,
                  eta = 0.1,
                  max_depth = 1,
                  nround=25, 
                  subsample = 0.5,
                  eval_metric = "auc",
                  objective = "binary:logistic")


mean(predict(basic_train,data.matrix(random_df)))

When I run this I get

[1] 0.04864751

Clearly the dataset I created has a mean of 1%. Why would the mean of probabilities predicted by this model equal 4%? Am I doing something wrong here or xgboost just doesn't handle class imbalance in binary classification well?

Just for comparison, use GBM

gbm_train = gbm.fit(random_df,pred

       ,shrinkage=0.1
       ,interaction.depth=1
       ,n.trees=25
       ,distribution='bernoulli')


mean(predict(gbm_train,random_df,n.trees=25,type='response'))

And you get the expected result!

[1] 0.01008938
$\endgroup$
  • $\begingroup$ Does the phenomenon persist if you add more trees or increase depth? Also, try setting subsample=1 (default), or try setting subsample=0.5 for the gbm to make the comparison fairer. $\endgroup$ – Alex R. Feb 20 '18 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.