6
$\begingroup$

From what I know, both of them are sequential learners and only the 1st tree in the sequence gets built on the data and all the following trees that get built are to correct the mistakes of previous tree, hence improving the performance or decreasing the bias.

subsample parameters in xgboost and lightgbm dictates the percentage of rows used per tree building.

So, with this context, if subsample is set to 0.75, first tree gets built with 75% of the data and all the following trees will focus on correcting mistakes. So, what happens to the remaining 25% of the data? will another set of sequential tress be built parallelly? or am I missing something here or got something wrong?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. and this will prevent overfitting. Subsampling will occur once in every boosting iteration. $\endgroup$
    – user2974951
    Commented May 27, 2022 at 8:18
  • 1
    $\begingroup$ yes, if subsample is set to 0.5, for the first boosting iteration, only 50% of the randomly sampled data will be used. i want to know how the remaining 50% of the data will be used? will it be in another parallel boosting iteration? or how? $\endgroup$
    – Naveen Reddy Marthala
    Commented May 27, 2022 at 8:50

2 Answers 2

Reset to default
1
$\begingroup$

That 25% of the data is unused by that learner (i.e. that iteration) of the XGBoost model (assuming subsample=0.75). This is normal, what they do via the subsample argument is to implement bagging by subsampling once in every boosting iteration. This means that, as you described, a portion of the data is not used by that specific base-learner during the $i$-th iteration. In the $i+1$-th iteration, sub-sampling of the whole dataset is performed once again.

Through bagging (or more formally bootstrap aggregation) we practically bootstrap our estimator and allow ourselves to have a more robust overall result - think of it as estimating the sample mean via bootstrapping; just the "mean" here is the "expected prediction for each item" in our sample instead of a single "expected prediction for the sample's central tendency".

And note that irrespective of the subsampling proportion we might use (e.g. 10%), we can always provide estimates for all (i.e. 100%) of our sample items. Some will be out-of-sample of course and some in-sample. In that way, we can also estimate error gradients for all our sample items if needed.

Improve this answer
$\endgroup$
9
  • $\begingroup$ thanks @usεr11852. your answer has created more questions than it answered, at my side. how many base-learners does, say, a run(.fit()) of xgboost will have and what parameter controls that? from, what i know, ONE base learner starts by fitting to the data and all the following train iterations will be fit on the previous one's residuals and this continues for num_iterations number of times. if there will be another fresh base-learner, fit again, with another randomly sub-sampled 75% of the data from previous base-learner, will that fit num_iterations number of times too? $\endgroup$
    – Naveen Reddy Marthala
    Commented May 28, 2022 at 7:20
  • 1
    $\begingroup$ Haha, sometimes that is the case. I am glad I can help. The number of base-learners is controlled by the num_round/num_iterations argument. In boosting, each iteration fits a new base learner. That base learner itself is (in the simple cases) a tree. There are no iterations involved in that tree fitting, just recursive partitioning that is controlled by the relevant parameters (e.g. subsample, max_depth, etc.). $\endgroup$
    – usεr11852
    Commented May 28, 2022 at 11:52
  • $\begingroup$ more confustion and more questions now. in my comment, by num_iterations, i meant the argument n_estimators. 1. C failed to find any parameter called num_round in xgboost and lightgbm's python API docs. 2. "In boosting, each iteration fits a new base learner.". in this, is 'iteration' the same thing as n_estimators? 2.1 if yes, will the num of base-learners be equal to n_estimators? $\endgroup$
    – Naveen Reddy Marthala
    Commented May 28, 2022 at 13:05
  • 1
    $\begingroup$ 1. The XGBoost docs are messed up at the moment the parameter obviously exists, the LightGBM ones defo have them just Control+F num_b. 2. Yes. 2.1. Yes. 3. Yes, the base learner. But the tree itself won't be "improved", the overall boosting ensemble performance will be improved. $\endgroup$
    – usεr11852
    Commented May 28, 2022 at 13:14
  • 2
    $\begingroup$ 1. The number of base learners built. 2. Forget the blogs. Read the relevant chapter (Chapt. 10) from Elements of Statistical Learning and ask questions about what you don't understand. 3. Correct. $\endgroup$
    – usεr11852
    Commented May 29, 2022 at 11:16
3
$\begingroup$

I found other answers are confusing. Here is my understanding. XGBoost trees estimate the residual from the previous tree (we set a base_score as the 0th tree residual). When subsample is defined (rather than default 1.0), each tree is trained using the subsampled data, and each tree computes the residuals of all data; this makes sure you have all residuals to subsample for consequent trees. n_estimators are the number of your trees; it is not a bad idea to think of it as the iteration number in DNN, because you can also early stop the training before all trees are trained using early_stopping_rounds hpyerparameter. Hope it helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.