Overfitting with parameter tuning I want to analyze data. To do so, I decided to try different models, with different parameters. But, I feel that if we try many models, just by pure chance some of them will have good results even on test and validation sets. How to organize parameter fitting and model selection so that we won't overfit data?
 A: 
I feel that if we try many models, just by pure chance some of them will have good results even on test and validation sets.

Yes, that is a valid concern.
It is not possible to give much more advise without knowing more about your data analysis problem. In particular, the number of cases available as well as number of features and the number of comparisons you plan to make. In general, model comparison needs large numbers of test cases.
You may be interested in reading our paper
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33. 
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323
which discusses (among some other points) factors influencing this random uncertainty and briefly discusses consequences for model comparison. These things are more difficult with classifiers than with regression models, but the general ideas hold for both: 


*

*there is random uncertainty due to the limited number of test cases

*as well as variance in the actual model's performance due to possible model instability

*Model instability depends mostly on the ratio of training samples : features, while testing uncertainty depends on the absolute number of test cases.

*You can treat the comparison as statistical testing of the hypothesis that one model is superior than the other. 


Note that in my field we are consistently hampered by very low case numbers - this is different in other fields. 

All that being said, as long as you reserve test data for an honest validation of the final model, you may decide to risk some overfitting and then do a rigorous test of the obtained model. Even if the model/hyperparameter optimization did not give you the optimal model, you then have a model at of the performance you measured.  
