Cross validation and model training (II)

Question

I am a newbie in ML, and I have a question about training the model with cross-validation similar to this one

In this problem, it was suggested to find the hyperparameters with GridSearchCV with 80% of the data, and then train the model with the same 80%.

In my case, I am using hyperoptimization library to find the hyperparameters with 80% of the data, and then I train the model with cross-validation KFold. Finally, I test the data.

1. Is this sequence correct?

2. If so, for each iteration during the cross-validation, I obtain the error of the model (R-Squared), and then I display an average of the errors. Is this correct, or I just should calculate the R-Squared with the test set?

Answer 1

It's a bit unclear from your question, but the correct sequence is:

1. Split your data into training data (80%) and test data (20%).
2. Split the training data into K "folds", typically 3 or 5.
3. Use cross validation to find the best hyperparameters. This means that you train your model on 4 of the folds, then compute error/loss on the remaining fold. Do this for each fold, so if you have 5 folds you repeat this process 5 times.
4. Train the model with the best hyperparameters on the entire training set, then compute error/loss on the test set.

So to answer your questions:

1. Yes, if I understand you correctly.
2. The former. Compute the error of the model on each iteration, using the held-out data from the K-fold CV procedure. Use the average across all K iterations. Do not use the test set during cross validation.