For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology.
I know that I would like to initially explore the parameter space for each model; varying parameters such as
learning_rate where applicable. Then selecting the best model parameters for each model by comparing performance (OOB/MSE where applicable) between the models in their respective parameter spaces. I would then end up with a set of parameters for each model with which I can use to validate and compare each models performance against the data set.
I have identified three options and I am not sure which I should choose.
Split the data into a training and testing set (90/10). Explore model parameter space on the training set. Identify the best model parameters. Retrain the model on the training set using the best parameters. Finally, calculate the test MSE by predicting the response values of the test set with the trained model.
Using all of the data explore the parameters space. Identify the best model parameters. Using the entire data set again, perform k-fold cross validation to calculate a final MSE for each model.
Split the data into a training and testing set (80/20). Explore model parameters space on the training set. Identify the best model parameters. Perform k-fold cross validation on a model using the best parameters trained on the entire data set (train + test), calculate a final test MSE for each model.
I really lean toward option 1 or 3. Option 2 seems to neglect the idea of calculating an MSE based on unseen data. Further option 1 seems the best between 1 and 3. Option 3 also somewhat neglects the idea of calculating an MSE based on unseen data; it would have some bias toward the training data used.
Is there a generally excepted methodology for the situation I described in the first part? I would assume that it is something very close to, if not, option 1?