I want to perform a manual features/parameters selection (Model Tuning) and, after that, perform a model evaluation with a cross-validation (k=10) on the entire dataset.
What is the best strategy? Intuitively, I would do those steps :
- Divide dataset in validation_train (60%), validation_test (20%) and test (20%).
- Process the Model Tuning on the validation_train and validation_test.
- Repeat (2) until a good accuracy is reached.
- Finally, process a CV-10fold on the entire dataset (validation_train, validation_test and test).
Or, this strategy conducts to a too optimistic score?
- Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation The Journal of Machine Learning Research, JMLR. org, 2010, 11, 2079-2107.
- N. Japkowicz and M. Shah, Evaluating learning algorithms: a classification perspective, Cambridge University Press, 2011