Model Tuning and Model Evaluation in Machine Learning Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task.
I want to perform a manual features/parameters selection (Model Tuning) and, after that, perform a model evaluation with a cross-validation (k=10) on the entire dataset.
What is the best strategy? Intuitively, I would do those steps :


*

*Divide dataset in validation_train (60%), validation_test (20%) and test (20%).

*Process the Model Tuning on the validation_train and validation_test.

*Repeat (2) until a good accuracy is reached.

*Finally, process a CV-10fold on the entire dataset (validation_train, validation_test and test).


Or, this strategy conducts to a too optimistic score?
Ref :


*

*Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation The Journal of Machine Learning Research, JMLR. org, 2010, 11, 2079-2107.

*N. Japkowicz and M. Shah, Evaluating learning algorithms: a classification perspective, Cambridge University Press, 2011

 A: I would have thought it was better to 


*

*Divide the dataset into training (80% or whatever) and testing (20% or whatever)

*Use $k$-fold cross-validation on the training set to tune your model

*Repeat (2) until you have optimised your model.  Your model is now tuned

*Use your model to predict on the testing set to get an estimate of out-of-model errors
You could consider putting your full dataset back into the model (without retuning), though you would not longer be able to make statements about out-of-model errors, and the impact of widening the inputs would usually be small.
A: Here is an approach for tuning.
Assume you have a dataset $D = {<x_1, c(x_1)>, <x_2, c(x_2)>, <x_3, c(x_3)>, <x_4, c(x_4)>, <x_5, c(x_5)>, <x_6, c(x_6)>}$ where $x_i$'s are examples and where $c(x_i)$'s are labels.


*

*Use examples from $x_1$ to $x_5$ as training data

*Generate some models using examples from $x_1$ to $x_4$

*

*Use different parameter values of the learning algorithm (e.g. use different $k$ values for $k$-NN such as 5, 10, 15)


*Validate (i.e. test) generated models over $x_5$ ($x_5$ is validation set)

*Repeat steps 2 and 3 to make cross validation (e.g. Generate models using examples $x_1$, $x_2$, $x_3$ and $x_5$, and validate these models using $x_4$)

*Once a final model is ready, test it using the test set, $x_6$

