I am trying gradient boosting on a dataset with event rate about 1% using Enterprise miner, but it is failing to produce any output. My question is, since it a decision tree based approach, is it even right to use gradient boosting with such low event?

  • 3
    $\begingroup$ You are dealing with imbalanced dataset. Boosting is indeed a good way to cope with it. For details see stats.stackexchange.com/questions/157940/… $\endgroup$ – DaL Mar 1 '16 at 7:32
  • $\begingroup$ But for me logistic regression is giving better results than randomforest or gradient boosting. I wanted to improve the performance of my model, by trying the boosted trees. $\endgroup$ – user2542275 Mar 1 '16 at 8:57
  • $\begingroup$ Boosting is based on weak classifiers. Theoretically, any weak classifier that is slightly better than random will do. In practice different algorithms are more suitable to some datas set so the weak classifier you choose is important. Can you specify more about the algorithms you used, their results and the data set? $\endgroup$ – DaL Mar 1 '16 at 9:13
  • $\begingroup$ Ok. About the dataset: Sample size>4m, event rate=1.2%. Number of predictors which are significant p-value<0.05 are 150. Logistic regression with most significant variables gave lift of 3 at 20% population. Neural network gave a lift of about 2.8. Gradient boosting did not produced any output, until i used stratified sampling with inverse prior weights. But the performance is poor. $\endgroup$ – user2542275 Mar 1 '16 at 10:31
  • $\begingroup$ Since your data set is quite big, you should have enough samples of your minority class, so the problem is due to relative imbalance. You have quite a few features but not too much, but indeed decision tree are less suitable for such datasets. I suggest that you'll create a balanced dataset and see how well your algorithms perform on it. Than you'll be able to apply the algorithm on the original dataset the way I described in the first comment. $\endgroup$ – DaL Mar 1 '16 at 11:32

(To give short answer to this:)

It is fine to use a gradient boosting machine algorithm when dealing with an imbalanced dataset. When dealing with a strongly imbalanced dataset it much more relevant to question the suitability of the metric used. We should potentially avoid metrics, like Accuracy or Recall, that are based on arbitrary thresholds, and opt for metrics, like AUCPR or Brier scoring, that give a more accurate picture - see the excellent CV.SE thread on: Why is accuracy not the best measure for assessing classification models? for more). Similarly, we could potentially employ a cost-sensitive approach by assigning different misclassification costs (e.g. see Masnadi-Shirazi & Vasconcelos (2011) Cost-Sensitive Boosting for a general view and proposed changes to known boosting algorithms or for a particular interesting application with a simpler approach check the Higgs Boson challenge report for the XGBoost algorithm; Chen & He (2015) Higgs Boson Discovery with Boosted Trees provide more details).

It is also worth noting that if we employ a probabilistic classifier (like GBMs) we can/should actively look into calibrating the returned probabilities (e.g. see Zadrozny & Elkan (2002) Transforming classifier scores into accurate multiclass probability estimates or Kull et al. (2017) Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers) to potentially augment our learner's performance. Especially when working with imbalanced data adequately capturing tendency changes might be more informative than simply labelling the data. To that extent, some might argue that cost-sensitive approaches are not that beneficial in the end (e.g. see Nikolaou et al. (2016) Cost-sensitive boosting algorithms: Do we really need them?). To reiterate the original point though, boosting algorithms are not inherently bad for imbalanced data and in certain cases they can offer a very competitive option.

| cite | improve this answer | |
  • $\begingroup$ I believe Brier scoring is equivalent to the Accuracy measure so will have the same limitations as Accuracy when assessing rare event models. $\endgroup$ – RobertF Feb 5 at 15:23
  • $\begingroup$ Brier score is not equivalent to Accuracy. Please note that we use the predicted probability for the calculation of the Brier score while for the Accuracy calculation we use labels based on hard thresholding of the predicted probabilities. $\endgroup$ – usεr11852 Feb 5 at 16:17
  • $\begingroup$ Thanks for clarifying - using the estimated probability rather than 0/1 for the predicted class makes more sense. $\endgroup$ – RobertF Feb 7 at 15:38
  • $\begingroup$ Cool. I am glad we sorted that out! :) $\endgroup$ – usεr11852 Feb 7 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.