Is there overfitting in this modellng approach

Question

I recently was told that the process I followed (component of a MS Thesis) could be seen as over-fitting. I am looking to get a better understanding of this and see if others agree.

The objective of this part of the paper is to

• Compare performance of Gradient Boosted Regression Trees against Random Forests on a data set.

• Look at the performance of the final model chosen (either GBM or RF).

The gbm and randomForest packages in R are being used, along with
caret.

The process followed was as follows:

• Preliminary pre-processing of the data (e.g. plugging missing values of the nominal predictors with a distinct category called "Missing"). The target variable was not looked at in regard to any pre-processing (which was very minimal).
• Create a grid of values for the meta-parameters of each algorithm (e.g. number of iterations for GBM).
• Create 25 random splits of the data set (65% training and 35% test).

Repeat 25 times the following for GBM (Each time utilizing one of the random train/test splits. Each time, which training and test set are "current" of course changes - this is repeated leave-group-out cross validation):

• Use 5-fold cross validation to find the "optimal" parameter settings of the algorithm over the grid search. Nothing from prior runs used at all in the current run.
• Once determined, fit a model to the full "current" training set and predict the "current" test set. Set aside the performance measure of this run.

Once 25 performance measures (actually a domain specific measure, but think of it as accuracy) are obtained in this fashion, follow the exact same process, using the exact same independent samples of train and test, for RF (same process, just with different grid search of course).

Now,I have 25 measures of performance from the then "current" test sets for GBM and RF. I compare them using a Wilcoxon Signed Rank Test and also a permutation test. I found GBM to be superior. I also claimed that the distribution of the performance measure from these 25 runs for GBM is the expected performance of the final GBM classifier.

What I did not do, was to pull out a random test set from the very beginning and set it aside to be compared to the final GBM model built from all the training data. I contend that what I did was actually much better as I repeated the split data / tune model / test on hold out process 25 times versus only once.

Is there over-fitting here? Since the 25 runs were used to select GBM versus RF does that mean that the performance measures acquired from the process can not be used as performance estimation for the full model?

EDIT In response to Wayne's comment, here is what was done during each of the 25 runs:

1. The data sampled for the ith training set (i=1,..,25) was broken up into 5 equally sized groups. A model was fit using 4 of the 5 groups, setting the GBM parameters (e.g. number of iterations) equal to the values in the jth (j=1,..,18) grid.
2. The performance on the 5th group was calculated using this model.
3. Steps 1 and 2 were repeated 4 more times (regular old k-fold CV with k=5). The performance was averaged from the 5 sub-runs and this made up the expected performance of GBM with that certain set of parameter values.
4. Steps 1 -3 were repeated for the 17 other "rows" in the grid.

Once completed, the best parameter values from the above exercise were determined and a GBM was fit using these parameter values and the full ith training set. Its performance was estimated on the ith test set.

Once this whole process was done 25 times, there were 25 measures of performance available for GBM. Then they were gathered for RF in the exact same way.

After comparing and choosing GBM, I looked at those 25 performance measures and took the mean and Stnd Error to determine a confidence interval for a GBM model on this data.

Answer 1

Imagine instead of just selecting GBM vs RF, you instead were choosing among 100 different GBM classifiers (assuming GBM training uses some kind of randomness, and you assign them random seeds 1 to 100). Then you'd pick one of those 100 GBMs as the best GBM. But it's almost certainly the case that the one of 100 models that you picked got lucky to beat its 99 siblings, and so your performance estimate will be optimistic.

You only used two learners, and they weren't trained by an identical algorithm, so your single GBM probably didn't have much selection pressure on it (especially if it dramatically outperformed the RF), but your error estimate is still going to be a little bit optimistic.

Answer 2

It sounds like you've tried mixing two techniques, each of which is legitimate, but the way you've done it feels like you'll end up with data leaking between cases.

At the lower level, you seem to be using CV correctly. Which would suggest that the top level should also be a CV, resulting in a nested CV, but your top level isn't CV.

At the top level, it sounds like you might've wanted to do a bootstrap validation, in which case a double-bootstrap would work for the top and lower levels, but your top level's not a proper bootstrap.

You might want to look at this stackexchange article. and the articles it links to, and perhaps restructure your top level to be a CV. Also, note that none other than Frank Harrell posts in that thread and suggests that CV be repeated a boatload of times. (I was shocked, thinking one repetition should do it.)