Does ensembling (boosting) cause overfitting?

I'm using SPSS Statistics Base 20. Using Analyze ==> Regression ==> Automatic Linear Modeling I've input about 50 variables.

When using no boosting, the reported accuracy of the model is 21%. The software does not tell me what is meant by accuracy, though.

When I then enable boosting and set the amount of ensembles to 400, it takes about half an hour to compute, and finally I get a model where all variables are used with an accuracy of about 70%.

Is it likely that this bump in accuracy is because of overfitting, or does this boosting technique really make my model more accurate?

-

First of all, let's make sure what you mean by overfitting. I assume you mean that the algorithm has learned too many of the nuances of the training data and will not perform well when you apply it to new data it hasn't seen before (from a similar population). This would also be known as poor generalization.

All machine learning algorithms, boosting included, can overfit. Of course, standard multivariate linear regression is guaranteed to overfit due to Stein's phenomena. If you care about overfitting and want to combat this, you need to make sure and "regularize" any algorithm that you apply. For regular regression, the simplest and often best method of regularization would be ridging.

For boosting specifically: to combat overfitting is usually as simple as using cross validation to determine how many boosting steps to take. On a more subtle level you probably want to make sure and use a small enough learning rate. Really small learning rates can take forever to overfit (take a ton of steps) so it's harder to screw them up. For pure accuracy though, you want to use as small of learning rate as you can and push the boosting steps right up until it does start to overfit, so if you really care you need to find the smallest learning rate that you can feasibly "bottom out". I believe gbm in R also bags a sample for each step, although I'm not sure that actually combats overfitting as much as it does spread the learning across the training data.

So, to your specific question, we can't really know if 400 ensembles is too many. In fact, the only way you really can is via Cross Validation or a hold out set (Or a kind of OOB estimate if your boosting algorithm does do the bagging at each step). If your base learner in each step is too strong or the learning rate is too high, then those 400 ensembles could easily be a drastic overfit. With no other data than a 21% to 70% gain, I would lean towards it overfitting.

-
Thanks. Unfortunately SPSS does not allow me to set all those settings such as learning rates. I'm not familiar with R either. I don't think SPSS can perform that cross validation test. I guess I should assume that a bump from 21% to 70% is too extreme. Thanks for your answer. –  Tom Jan 7 '12 at 14:26
You can manually perform a hold-out validation I'm sure. Just sample out 20-30% of your data and don't train on that portion. Then compute your chosen error metric on that hold out set. It's a little more inefficient than full Cross Validation, but it still is a lot better than nothing. –  Shea Parkes Jan 7 '12 at 15:08
Okay, the trained set was 80% correct while my sub-sample was only estimated correctly 40% of the times. This indicates extreme overfitting if I am correct. Thanks a lot for your help, though the outcome is disappointing. I thought I had a working model :( –  Tom Jan 7 '12 at 18:05
So, I didn't go into it above, but I pesonnally define "overfitting" and "generalizing" to mean two different things. I usually think of your model as "overfit" because it has a much higher error rate on new data. However, I would view it as generalizing well still since it looks like it has double the accuracy of your base linear model (21% vs 40%). I'd just run it again at maybe half the number of boosting steps and test it again. Purists would suggest you make a third sample of your data so you can report a final "true" error rate after you use the first sample to tune your #steps. –  Shea Parkes Jan 7 '12 at 19:07