I understand that we need to split our data into training, validation and test set - we use training set to train the model, and use cross validation on the validation set to tune the model, and finally, we want to use the set aside testing set that's never seen by the model to get an honest representation of the generalized performance on population or unseen data.
However, I am not sure whether we should further tune our model after getting a test result on the test set, in particular, whether to optimize the model based on the test result based on the test set.
My understanding is, if we see a certain parameter setting seems to lift the test set performance, and as a result we use that parameter setting vs. the previous one, this is "data leakage" - giving knowledge to the model of the data, resulting in overfitting.
On the other hand, if we don't do anything, it doesn't make any sense to use the test set for more than once then, maybe just one single time at the end of everything in terms of model building and evaluation. But again, what if the performance is really bad based on the test set, do we not go back to the model and further test other parameter combinations not already used in cross-validation? If we do, it seems we're coming back to the question in the previous paragraph, we're overfitting...really struggling with this process.
Also, it seems these two posts suggest different solutions. The first post indicates we could further tune based on test result. However, the second post here clearly says we should do nothing! But, again, if we do nothing, that implies we can only use the test set ONE TIME...
Can someone please help me clarify these concerns? Thanks in advance!