I have a question about cross-validation and hyperparameter optimization. Namely about how to test the final model performance in an unbiased way.
Now, I know I have to train the model, optimize the hyperparameters and test the model on separate datasets. But I'm wondering if I can first optimize the hyperparameters on some validation data, then resample my dataset to create new train and test sets and evaluate the model with the chosen hyperparameters? Specifically, I have a small dataset and a neural net model which is proving difficult to train and optimize. The models performance appears to vary quite a bit based on the train-test splits, so I want to use 5-fold cross validation. The procedure I'm thinking of doing is:
- Do 5-fold cross validation on the entire dataset, to optimize the hyperparameters.
- Split the dataset into a new train and test set
- Train a model on the train set using the best hyperparameters from step 1. Evaluate on the test set.
The question is, will this give unbiased estimates of the error, since the model is new and these exact train-test sets were never used in hyperparameter optimization? Or will the error be underestimated, because the same datapoints were used in original cross-validation, so that the hyperparameter might have overfit to this data?