How to estimate the error of the model in the best way?

Question

I have many samples in my data and I want to fit many models which should be tuned by bayesian optimization. What is the best way for estimation of error, in statistical point of view? I mean, some estimates are biased, some have too high variance, I want to find some good way. I have some ideas and I would appreciate a discussion:

1) Split train data to train and validation set and use it in this setting for all estimations of hyperparameters and model selection.

2) Split the data ti training and validation set each time randomly when I will use new method, but keep the setting during hyperparameter tunning.

3) Split the data randomly to train and validation set in each step of bayesian optimization, for each method (this should lead to low bias probably, as the results will not be biased only towards the only split as mentioned in 1) )

4) Do the cross validation in a way, that I split the data to $k$ folds once in the beginning and do the cross validation in bayesian optimization and when the model is selected, I can train the final model on the whole training set

5) Split the data to $k$ folds in each iteration of bayesian optimization, i.e. compute CV-error with current parameters, choose new parameters, split the data randomly to $k$ folds again and compute CV-error with current parameters etc... This should lead to lower bias towards the split ?

Or any other idea? I would really appreciate also some justification and some discussion, I assume there is no "general" answer, but everything has its pluses and minuses. Or some good literature, what I have found so far does not answer my question. Thanks to whoever is interested !

Answer 1

This depends on how big is the dataset and how much time and computational resources you have. If you can afford training a model many times I'd do option 5, i.e.

For each vector of hyperparameters $h \in H$ perform a $k-$fold Cross Validation to estimate the performance of the model. After some predefined number of iterations (e.g. 20) train the model again using the best found vector of hyperparameters $h^*$ with the whole dataset.

However, since you've mentioned

I have many samples in my data

and if you're planning to train some deep neural network, then you're likely cannot afford using Cross Validation, since training is quite expensive. In such case I'd do option 1, i.e.

Split your dataset $X$ into training $X_{train}$ and validiation $X_{val}$ sets (e.g. in proportions $80:20$) and then, for each vector of hyperparameters $h$ train a model and validate it using $X_{val}$. At the end, select the best $h^*$ and retrain the model again using the whole $X$.

Note that if you're doing research and you want to explicitly measure and report the performance of your model, you will likely need to split your dataset $X$ into three sets: $X_{train}$ for training, $X_{val}$ for validation (hyperparameter tuning, i.e. Bayesian Optimisation) and $X_{test}$ which you can test your model on and report its performance.