I was reading some questions and answer about the reasons and differences for the split of the datasets in Train, Validation and Test Set. I came up with two questions that I'm not completely sure about the answers, and I will use as an example for the questions a polynomial regressor with D = degree of my polynomial.
Validation Set is only necessary when we have hyperparameters in our model, otherwise validation is useless. Suppose that I can only train a quadratic polynomial, i.e. D = 2, I cannot choose D=1, D=3 or other values. Whats the point here in splitting the Training Set ( and so having less data to train my model) evaluate it using the Validation Set and then evaluate it again but this time using the Test Set? Seems like we are doing the same thing two times in a row. I only see the sense in using a Validation Set when we need to tune the hyperparameters of the model, make a lot of "pre test", one for each value of D, choose the best one and in the end test our final model (suppose D =3) using the Test Set. Does this make sense?
Why its bad to use Training and Test Set multiple times? Using validation we should Train and Validate every time for each value of D, and finally test our final model (again, suppose D = 3) using the Test Set. Why its a bad idea to use the whole Training Set (no split in Validation) and Test Set for every value of D? For every value of D we train using the whole Training Set, and then test using the Test Set. What is going to tell us the Validation Set that the Test Set cannot tell?
Reason for K-fold Cross Validation Under the reasons for using K-fold Cross Validation instead of a simple Validation there is that if the Validation Set is not big enough we may risk to overfit the Validation Set. Shouldn't be the Training Set that we risk overfit? We are using the Validation set only for evaluation, not for training, so why the risk to overfit it instead of the Training Set?