# Understanding over-fitting

I am trying to understand over-fitting. I am using a regression tree method in Matlab. The given sample size is 500, which I divide into a training set of 400 and a test set of 100. I create the model for the training set and get a corresponding $R^2$ of about 84%. i.e. the in-sample $R^2$ is 84%.

When I use the model to forecast for the 100 in the test set, I get a negative $R^2$ which indicates a very poor model on out-of-sample data.

So, does it mean that my model over-fitted the in-sample data of size 400 and got a very high R-square? The regression tree in Matlab has pruning turned on by default so that should avoid over-fitting.

Any ideas as to what this situation might mean ?

• Can you provide some more details? How many potential IVs are there in the tree? Did you randomly separate the training and test set? @FrankHarrell has posted on trees, indicating they can be very unstable without very large N. I hope he sees this, then he can provide details of those issues. – Peter Flom Oct 7 '13 at 12:35
• Can you add some matlab code for your issue? Usually matlab offers a way to use cross-validation within a classifier rather than just one train/test run. You should also check that test and train sets are randomly selected rather than referring to one portion of dataset – BGreene Oct 7 '13 at 13:06

If in-sample $R^2$ is $.84$ and out-of-sample $R^2$ is negative, then you are overfitting the data. The fact that prune is turned on doesn't automatically mean that overfitting will be avoided.