The bag.fraction
parameter in SGB controls the size of the random subsample of the original training set on which each successive weak learner is fitted:
At each iteration a subsample of the training data is drawn at random (without replacement) from the full training data set. This randomly selected subsample is then used, instead of the full sample, to fit the base learner and compute the model update for the current iteration.
(Quote from the original SGB paper by Friedman).
What I don't get is how this is compatible with the fact that in Gradient Boosting, each successive base model is fitted directly on the residuals (or gradient of the loss function) of the current model.
In other words, how can we fit each new base model on the residuals of the current model and at the same time on a random subsample of the original training set? What is the link between these two key steps?
If someone with enough Python literacy could find the information from the code, that'd be great. Also, the R code is not useful as it is only a wrapper for a C implementation.
Boosting experts who frequent our site will already be well aware of your question
, it seems that (2) may actually be true; which is actually rather depressing. $\endgroup$ – Antoine Oct 7 '15 at 17:25