# Does ensembling (boosting) cause overfitting?

I'm using SPSS Statistics Base 20. Using Analyze ==> Regression ==> Automatic Linear Modeling I've input about 50 variables.

When using no boosting, the reported accuracy of the model is 21%. The software does not tell me what is meant by accuracy, though.

When I then enable boosting and set the amount of ensembles to 400, it takes about half an hour to compute, and finally I get a model where all variables are used with an accuracy of about 70%.

Is it likely that this bump in accuracy is because of overfitting, or does this boosting technique really make my model more accurate?

-

For boosting specifically: to combat overfitting is usually as simple as using cross validation to determine how many boosting steps to take. On a more subtle level you probably want to make sure and use a small enough learning rate. Really small learning rates can take forever to overfit (take a ton of steps) so it's harder to screw them up. For pure accuracy though, you want to use as small of learning rate as you can and push the boosting steps right up until it does start to overfit, so if you really care you need to find the smallest learning rate that you can feasibly "bottom out". I believe gbm in R also bags a sample for each step, although I'm not sure that actually combats overfitting as much as it does spread the learning across the training data.