I have a separate training and test set. How can I use k-fold cross validation to train a model?

Question

I have a training set, and a test set. I think it is important for the model to be tested always on the same test set. Therefore, I cannot mix the training and test sets and perform k-fold cross validation on the whole dataset. (please correct me if I am wrong)

Nevertheless, I need to tune the parameters of my model. I have seen many papers saying, "we tuned the parameters using k-fold cross validation". I know that if I have a single validation set, I can use that to tune the data and then report the results for the test set. How about k-fold cross validation? Let's say k=5. I will have 5 different models and 5 different final results on my test set.

Now I have two questions:

1. If I want to report the overall result of my model on the test set, should I average the performance of those 5 models and report it? Or should I pick one of them?
2. If I want to pick a final parameter set (or trained model). Should I pick one of them? Or average (if possible) them and have a final parameter set? If the averaging is done, then I can report the final result on the test set using the averaged parameter set, right?

Answer 1

I'm adding this as an answer as the discussion was getting too long for chat.

Your original question has to do about how to use k-fold cross-validation. As the name suggests, it is a technique for validation of a specific model.

If as you require you want to combine the five models to produce an "average" model this is independent of k-fold and moves into the realm of ensemble models.

Use k-fold to assess stability of your model based on standard deviation.

Build a model using the entire training set.

Predict on test set and assess model performance.

The question of reporting "average performance" is a personal choice. There is no right or wrong approach, however doing so you would be advised on researching ensemble models instead of k-fold cross validation. In fact, you could end up doing a k-fold cross validation of an ensemble model too!