I recently was told that the process I followed (component of a MS Thesis) could be seen as over-fitting. I am looking to get a better understanding of this and see if others agree.
The objective of this part of the paper is to
Compare performance of Gradient Boosted Regression Trees against Random Forests on a data set.
Look at the performance of the final model chosen (either GBM or RF).
randomForest packages in R are being used, along with
The process followed was as follows:
- Preliminary pre-processing of the data (e.g. plugging missing values of the nominal predictors with a distinct category called "Missing"). The target variable was not looked at in regard to any pre-processing (which was very minimal).
- Create a grid of values for the meta-parameters of each algorithm (e.g. number of iterations for GBM).
- Create 25 random splits of the data set (65% training and 35% test).
Repeat 25 times the following for GBM (Each time utilizing one of the random train/test splits. Each time, which training and test set are "current" of course changes - this is repeated leave-group-out cross validation):
- Use 5-fold cross validation to find the "optimal" parameter settings of the algorithm over the grid search. Nothing from prior runs used at all in the current run.
- Once determined, fit a model to the full "current" training set and predict the "current" test set. Set aside the performance measure of this run.
Once 25 performance measures (actually a domain specific measure, but think of it as accuracy) are obtained in this fashion, follow the exact same process, using the exact same independent samples of train and test, for RF (same process, just with different grid search of course).
Now,I have 25 measures of performance from the then "current" test sets for GBM and RF. I compare them using a Wilcoxon Signed Rank Test and also a permutation test. I found GBM to be superior. I also claimed that the distribution of the performance measure from these 25 runs for GBM is the expected performance of the final GBM classifier.
What I did not do, was to pull out a random test set from the very beginning and set it aside to be compared to the final GBM model built from all the training data. I contend that what I did was actually much better as I repeated the split data / tune model / test on hold out process 25 times versus only once.
Is there over-fitting here? Since the 25 runs were used to select GBM versus RF does that mean that the performance measures acquired from the process can not be used as performance estimation for the full model?
EDIT In response to Wayne's comment, here is what was done during each of the 25 runs:
- The data sampled for the ith training set (i=1,..,25) was broken up into 5 equally sized groups. A model was fit using 4 of the 5 groups, setting the GBM parameters (e.g. number of iterations) equal to the values in the jth (j=1,..,18) grid.
- The performance on the 5th group was calculated using this model.
- Steps 1 and 2 were repeated 4 more times (regular old k-fold CV with k=5). The performance was averaged from the 5 sub-runs and this made up the expected performance of GBM with that certain set of parameter values.
- Steps 1 -3 were repeated for the 17 other "rows" in the grid.
Once completed, the best parameter values from the above exercise were determined and a GBM was fit using these parameter values and the full ith training set. Its performance was estimated on the ith test set.
Once this whole process was done 25 times, there were 25 measures of performance available for GBM. Then they were gathered for RF in the exact same way.
After comparing and choosing GBM, I looked at those 25 performance measures and took the mean and Stnd Error to determine a confidence interval for a GBM model on this data.