# Does ensembling (boosting) cause overfitting?

I'm using SPSS Statistics Base 20. Using Analyze $\rightarrow$ Regression $\rightarrow$ Automatic Linear Modeling I've input about 50 variables.

When using no boosting, the reported accuracy of the model is 21%. The software does not tell me what is meant by accuracy, though.

When I then enable boosting and set the amount of ensembles to 400, it takes about half an hour to compute, and finally I get a model where all variables are used with an accuracy of about 70%.

Is it likely that this bump in accuracy is because of overfitting, or does this boosting technique really make my model more accurate?

First of all, let's make sure what you mean by overfitting. I assume you mean that the algorithm has learned too many of the nuances of the training data and will not perform well when you apply it to new data it hasn't seen before (from a similar population). This would also be known as poor generalization.

All machine learning algorithms, boosting included, can overfit. Of course, standard multivariate linear regression is guaranteed to overfit due to Stein's phenomena. If you care about overfitting and want to combat this, you need to make sure and "regularize" any algorithm that you apply. For regular regression, the simplest and often best method of regularization would be ridging.

For boosting specifically: to combat overfitting is usually as simple as using cross validation to determine how many boosting steps to take. On a more subtle level you probably want to make sure and use a small enough learning rate. Really small learning rates can take forever to overfit (take a ton of steps) so it's harder to screw them up. For pure accuracy though, you want to use as small of learning rate as you can and push the boosting steps right up until it does start to overfit, so if you really care you need to find the smallest learning rate that you can feasibly "bottom out". I believe gbm in R also bags a sample for each step, although I'm not sure that actually combats overfitting as much as it does spread the learning across the training data.

So, to your specific question, we can't really know if 400 ensembles is too many. In fact, the only way you really can is via Cross Validation or a hold out set (Or a kind of OOB estimate if your boosting algorithm does do the bagging at each step). If your base learner in each step is too strong or the learning rate is too high, then those 400 ensembles could easily be a drastic overfit. With no other data than a 21% to 70% gain, I would lean towards it overfitting.

• Thanks. Unfortunately SPSS does not allow me to set all those settings such as learning rates. I'm not familiar with R either. I don't think SPSS can perform that cross validation test. I guess I should assume that a bump from 21% to 70% is too extreme. Thanks for your answer. – Tom Jan 7 '12 at 14:26
• You can manually perform a hold-out validation I'm sure. Just sample out 20-30% of your data and don't train on that portion. Then compute your chosen error metric on that hold out set. It's a little more inefficient than full Cross Validation, but it still is a lot better than nothing. – Shea Parkes Jan 7 '12 at 15:08
• Okay, the trained set was 80% correct while my sub-sample was only estimated correctly 40% of the times. This indicates extreme overfitting if I am correct. Thanks a lot for your help, though the outcome is disappointing. I thought I had a working model :( – Tom Jan 7 '12 at 18:05
• So, I didn't go into it above, but I pesonnally define "overfitting" and "generalizing" to mean two different things. I usually think of your model as "overfit" because it has a much higher error rate on new data. However, I would view it as generalizing well still since it looks like it has double the accuracy of your base linear model (21% vs 40%). I'd just run it again at maybe half the number of boosting steps and test it again. Purists would suggest you make a third sample of your data so you can report a final "true" error rate after you use the first sample to tune your #steps. – Shea Parkes Jan 7 '12 at 19:07

The overfitting of boosting techniques is a topic that is not yet theoretically understood, but empirically results show that boosting seems to be very robust against overfitting.

The usual explanation for this phenomena is as follow: the samples that are incorrectly predicted in one iteration will have higher weight in the next one. Thus, isolated and mislabelled points tend to strongly force the classifier to create complicated hypothesis to fit them, which we will call overfitting. However, as the hypothesis is very non-linear due to the combination of several classifiers (ensembles), the hypothesis around those problematic points is so narrow that is practically impossible that another point lies there.

For example, in the below example, we can see one data points that is mislabelled. The generated hypothesis (green line) will create a narrow circle around this point, but it is very unlikely that another point in the test data lies exactly in the same position.

Quoting Patrick H. Winston lecture of Artificial Intelligence in the MIT: This [boosting] doesn't seem to overfit. That is an experimental result for which the literature is confused respect to providing an explanation. So, this stuff has been tried in all sort of problems like handwriting recognition, understanding speech, all sort of stuff uses boosting, and unlike other methods for some reason, and yet imperfectly understood, it does not seem to overfit. [...] In conclusion, this is magic, you always want to use it and it will work with any classifier.

I think a good way to see if you are overfitting is to see the agreement between the individual nodes in the ensemble. If you get a very high agreement of 95%, there is high chance that your model may be predicting a very low accuracy of the target variable. This would definitely be a high indicator that your model has been overtrained and is unable to generalize well.