1
$\begingroup$

I am trying to understand over-fitting. I am using a regression tree method in Matlab. The given sample size is 500, which I divide into a training set of 400 and a test set of 100. I create the model for the training set and get a corresponding $R^2$ of about 84%. i.e. the in-sample $R^2$ is 84%.

When I use the model to forecast for the 100 in the test set, I get a negative $R^2$ which indicates a very poor model on out-of-sample data.

So, does it mean that my model over-fitted the in-sample data of size 400 and got a very high R-square? The regression tree in Matlab has pruning turned on by default so that should avoid over-fitting.

Any ideas as to what this situation might mean ?

$\endgroup$
2
  • $\begingroup$ Can you provide some more details? How many potential IVs are there in the tree? Did you randomly separate the training and test set? @FrankHarrell has posted on trees, indicating they can be very unstable without very large N. I hope he sees this, then he can provide details of those issues. $\endgroup$ – Peter Flom Oct 7 '13 at 12:35
  • $\begingroup$ Can you add some matlab code for your issue? Usually matlab offers a way to use cross-validation within a classifier rather than just one train/test run. You should also check that test and train sets are randomly selected rather than referring to one portion of dataset $\endgroup$ – BGreene Oct 7 '13 at 13:06
2
$\begingroup$

If in-sample $R^2$ is $.84$ and out-of-sample $R^2$ is negative, then you are overfitting the data. The fact that prune is turned on doesn't automatically mean that overfitting will be avoided.

For regression trees -- in fact, for any learning algorithm -- there are going to be a bunch of parameters to tune. I'm not familiar with Matlab's implementation of regression trees, but there must be at least one parameter that governs when the algorithm decides to stop splitting tree nodes. And then during pruning there is at least one parameter that determines when the tree stops getting pruned. Try to adjust those parameters and see if you can get better out of sample performance.

Of course, it might be the case that your data just doesn't fit a regression tree model very well, in which case you won't get a very good fit no matter what you do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.