Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter tuning or else we may end up overfitting the test set and have unrealistic results. So it's clearly wrong if we tested some model then try to change its hyperparameters and train, validate, and test it again on the same test set.
However, what if we trained and validated, for example, 5 different models, and we decided that we won't modify any of them again. Then we tested each of the 5 models on the (same) unseen test set. If we select the model that achieves the best test result, isn't this the same as if we try different hyperparameter combinations and select the one that has the best performance on the test set?
In this sense, is it wrong when research papers propose multiple methods, test them on the same data, and decide that one of them is the best because it has the best performance on the test set? Isn't it supposed that the test set doesn't influence the choice of the best method?
But if this is wrong, how are we supposed to compare methods (from different papers) on the same test set without being biased? I feel there is a contradiction regarding this point.