Yes, if you will repeat this process multiple times you will overfit the test set.
It happens in many Kaggel competitions when the group submit their results multiple times and try to tweak their models to produce better results on the public test set. Then when the competition ends and the private test set scores are calculated, their score drops dramatically. This is because their model overfits the public test set and can't generalize to new datasets.
A better approach is first split to train and test. Then derive your conclusion as much as you can on the train set. Means:
Perform nested-cross validation on the train set to hypertune parameters and to evaluate your models (inner loop - hyperparameter tunning, outer loop - model selection). Notice that the same model with different hyperparameters is considered as two different models for the model selection phase.
For the 1-3 best models:
Analyze the difference in the results between the inner loop to the outer loop. Treat the outer loop scores as the test scores and the inner loop as the validation score.
*If the mean error of the inner loop is not satisfactory for you (compare it to the state-of-the-art or to human level), you have a bias problem. You'll want to build a more complex model (more trees, more layers in NN, train NN longer), feature engineering, feature selection.
*If the error std of the outer loop is high, or the mean error of the outer loop is higher than the inner loop means you have a variance problem. Your model is not stable and you may overfit to noise. You'll want to build a simpler model (i.e fewer trees in RF,
simpler NN), add more data, regularization, check if splits are representative enough, etc.
- Fix these issues and do these steps again and again until you are satisfied with the results. Only then, when you think there is nothing else to do, fit the model over all the train data and predict on the test data.
- Now, if you are not pleased with the results, you should do error analysis, understand what is missing in your model (more\less features, more\less complexity) and you are back to the drawing board. Start this all process all over again.
The best scenario will have a new test set in this case. I know it's not always available, that's why you should invest most of your time in the nested CV results analysis and improvements and delay predicting on the test set as much as you can.