I have a training set, and a test set. I think it is important for the model to be tested always on the same test set. Therefore, I cannot mix the training and test sets and perform k-fold cross validation on the whole dataset. (please correct me if I am wrong)
Nevertheless, I need to tune the parameters of my model. I have seen many papers saying, "we tuned the parameters using k-fold cross validation". I know that if I have a single validation set, I can use that to tune the data and then report the results for the test set. How about k-fold cross validation? Let's say k=5. I will have 5 different models and 5 different final results on my test set.
Now I have two questions:
- If I want to report the overall result of my model on the test set, should I average the performance of those 5 models and report it? Or should I pick one of them?
- If I want to pick a final parameter set (or trained model). Should I pick one of them? Or average (if possible) them and have a final parameter set? If the averaging is done, then I can report the final result on the test set using the averaged parameter set, right?