How to validate a model when first exploring model hyperparameter space?

Question

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology.

I know that I would like to initially explore the parameter space for each model; varying parameters such as n_estimators, max_features, learning_rate where applicable. Then selecting the best model parameters for each model by comparing performance (OOB/MSE where applicable) between the models in their respective parameter spaces. I would then end up with a set of parameters for each model with which I can use to validate and compare each models performance against the data set.

---

I have identified three options and I am not sure which I should choose.

1. Split the data into a training and testing set (90/10). Explore model parameter space on the training set. Identify the best model parameters. Retrain the model on the training set using the best parameters. Finally, calculate the test MSE by predicting the response values of the test set with the trained model.

2. Using all of the data explore the parameters space. Identify the best model parameters. Using the entire data set again, perform k-fold cross validation to calculate a final MSE for each model.

3. Split the data into a training and testing set (80/20). Explore model parameters space on the training set. Identify the best model parameters. Perform k-fold cross validation on a model using the best parameters trained on the entire data set (train + test), calculate a final test MSE for each model.

---

I really lean toward option 1 or 3. Option 2 seems to neglect the idea of calculating an MSE based on unseen data. Further option 1 seems the best between 1 and 3. Option 3 also somewhat neglects the idea of calculating an MSE based on unseen data; it would have some bias toward the training data used.

Is there a generally excepted methodology for the situation I described in the first part? I would assume that it is something very close to, if not, option 1?

Answer 1

If I understand your question correctly, you want find optimal hyperparameters for every method, and then compare the optimized models, without picking a "winner".

If you would want to just end up with a single "winning" model, tuning the hyperparameters and choosing between the models should be done in one sweep: Choosing between models and finding the best hyperparameters is qualitatively the same, namely selection of the best model. This can be done with nested cross-validation, which has extensively been discussed here.

When evaluating models, it is essential to never (even implicitly) mix training and test sets. There are excellent contributions by @Dikran Marsupial regarding this, often subtle, issue.

In case you indeed want to compare the different, optimized models, let's look at your 3 options:

1. This makes sense. However, if you do not combine it with a CV, you won't know much about the variation of the test MSE, and comparison of different test results will not really be comparable. Here I would use nested CV for every model, to be able to compare the performances. See the contribution mentioned above. Keep in mind, that nested CV aims to evaluate how good your hyperparameter-search is (i.e., the model selection), and not how good you model is (see the references above).
2. This is not recommendable. Since you optimized your hyperparameters on the entire set, any validation is bound to have a positive bias, since any test set is part of the data that has already been used for parameter tuning.
3. Also here you should expect a positive bias: The choice of the optimal hyperparameters was made by using the training set. When you do CV to evaluate the model on the entire data, the test folds will contain training data that was used to find the optimal parameters. Parts of the test data has therefore already been polluted: It has been used to optimize the parameters.

You criticized Options 2 and 3 for the right reasons, they will be biased. Option 1 makes sense, but I would recommend "enhancing" it and performing nested CV instead, and compare the models then. If you are interested in simply finding the best (or a best) model among all options, combine the parameter search and model selection, and still do nested CV.