Timeline for how does `subsample` parameter work in boosting algorithms like xgboost and lightgbm?
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 29, 2022 at 11:58 | comment | added | Naveen Reddy Marthala | Sure @usεr11852, I will read that chapter. thanks for your time and all the answers to my questions in comments. | |
| May 29, 2022 at 11:16 | comment | added | usεr11852 | 1. The number of base learners built. 2. Forget the blogs. Read the relevant chapter (Chapt. 10) from Elements of Statistical Learning and ask questions about what you don't understand. 3. Correct. | |
| May 29, 2022 at 6:40 | vote | accept | Naveen Reddy Marthala | ||
| May 29, 2022 at 6:37 | comment | added | Naveen Reddy Marthala |
3. And the number of base-learners fit will be decided by the parameter n_estimators. and each such base-learner will be fit with randomly(or by other sampling method that user chooses) samples subsample percentage of the data, correct?
|
|
| May 29, 2022 at 6:37 | comment | added | Naveen Reddy Marthala |
1. in lightgbm docs, num_iterations, num_trees, num_boost_round, n_estimators are the same thing and explanation on this parameter is "number of boosting iterations". so, is that the number of trees(or base-learners) that get built? or the improvements in overall boosting? 2. thanks for busting the myth. i have been under the assmption that the tree itself will be improved, the splits of tress, binning for a leaf etc,. will you be able tell me how "the overall boosting ensemble performance will be improved" or point me to any blog/resource that may help me learn more about this?!
|
|
| May 28, 2022 at 13:14 | comment | added | usεr11852 |
1. The XGBoost docs are messed up at the moment the parameter obviously exists, the LightGBM ones defo have them just Control+F num_b. 2. Yes. 2.1. Yes. 3. Yes, the base learner. But the tree itself won't be "improved", the overall boosting ensemble performance will be improved.
|
|
| May 28, 2022 at 13:05 | comment | added | Naveen Reddy Marthala | 3. "There are no iterations involved in that tree fitting,". which tree? the base-learner? yes, i meant, this tree will be sequentially improved by fitting to it's residuals in the following iterations. or i may have got this wrong. | |
| May 28, 2022 at 13:05 | comment | added | Naveen Reddy Marthala |
more confustion and more questions now. in my comment, by num_iterations, i meant the argument n_estimators. 1. C failed to find any parameter called num_round in xgboost and lightgbm's python API docs. 2. "In boosting, each iteration fits a new base learner.". in this, is 'iteration' the same thing as n_estimators? 2.1 if yes, will the num of base-learners be equal to n_estimators?
|
|
| May 28, 2022 at 11:52 | comment | added | usεr11852 |
Haha, sometimes that is the case. I am glad I can help. The number of base-learners is controlled by the num_round/num_iterations argument. In boosting, each iteration fits a new base learner. That base learner itself is (in the simple cases) a tree. There are no iterations involved in that tree fitting, just recursive partitioning that is controlled by the relevant parameters (e.g. subsample, max_depth, etc.).
|
|
| May 28, 2022 at 7:20 | comment | added | Naveen Reddy Marthala |
thanks @usεr11852. your answer has created more questions than it answered, at my side. how many base-learners does, say, a run(.fit()) of xgboost will have and what parameter controls that? from, what i know, ONE base learner starts by fitting to the data and all the following train iterations will be fit on the previous one's residuals and this continues for num_iterations number of times. if there will be another fresh base-learner, fit again, with another randomly sub-sampled 75% of the data from previous base-learner, will that fit num_iterations number of times too?
|
|
| May 27, 2022 at 9:04 | history | edited | usεr11852 | CC BY-SA 4.0 |
added 377 characters in body
|
| May 27, 2022 at 8:59 | history | edited | usεr11852 | CC BY-SA 4.0 |
added 377 characters in body
|
| May 27, 2022 at 8:51 | history | answered | usεr11852 | CC BY-SA 4.0 |