Model Tuning and Model Evaluation in Machine Learning

Question

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task.

I want to perform a manual features/parameters selection (Model Tuning) and, after that, perform a model evaluation with a cross-validation (k=10) on the entire dataset.

What is the best strategy? Intuitively, I would do those steps :

1. Divide dataset in validation_train (60%), validation_test (20%) and test (20%).
2. Process the Model Tuning on the validation_train and validation_test.
3. Repeat (2) until a good accuracy is reached.
4. Finally, process a CV-10fold on the entire dataset (validation_train, validation_test and test).

Or, this strategy conducts to a too optimistic score?

Ref :

• Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation The Journal of Machine Learning Research, JMLR. org, 2010, 11, 2079-2107.
• N. Japkowicz and M. Shah, Evaluating learning algorithms: a classification perspective, Cambridge University Press, 2011

Answer 1

I would have thought it was better to

1. Divide the dataset into training (80% or whatever) and testing (20% or whatever)

2. Use $k$-fold cross-validation on the training set to tune your model

3. Repeat (2) until you have optimised your model. Your model is now tuned

4. Use your model to predict on the testing set to get an estimate of out-of-model errors

You could consider putting your full dataset back into the model (without retuning), though you would not longer be able to make statements about out-of-model errors, and the impact of widening the inputs would usually be small.

Answer 2

Here is an approach for tuning.

Assume you have a dataset $D = {<x_1, c(x_1)>, <x_2, c(x_2)>, <x_3, c(x_3)>, <x_4, c(x_4)>, <x_5, c(x_5)>, <x_6, c(x_6)>}$ where $x_i$'s are examples and where $c(x_i)$'s are labels.

1. Use examples from $x_1$ to $x_5$ as training data
2. Generate some models using examples from $x_1$ to $x_4$
   1. Use different parameter values of the learning algorithm (e.g. use different $k$ values for $k$-NN such as 5, 10, 15)
3. Validate (i.e. test) generated models over $x_5$ ($x_5$ is validation set)
4. Repeat steps 2 and 3 to make cross validation (e.g. Generate models using examples $x_1$, $x_2$, $x_3$ and $x_5$, and validate these models using $x_4$)
5. Once a final model is ready, test it using the test set, $x_6$