I'm very confused about nested cross validation works . I have seen
that nested cross validation is necessary to split parameter selection and model selection, since using the test set to both select the values of the parameter and evaluate the model, I risk optimistically biasing my model evaluations. But it is not clear how exactly it works. I would like to confirm that my way of proceeding is correct, or that some people explain to me how to proceed.
Actually, I do so:
- Split the traing set int 3 parts: training set, validation set and test set
- I do a cross validation (5-fold-validation) on training and validation set.In this way for each parameter I have an extent that allows me choice the best parameters. Namely in this loop I find the best parameters(thus a model)
- And now model selection. Alone for the best parameters found I have to measure the accuracy. I have to consider the training set, the test set and the validation set and do another 5-fold-validation to find a reliable measure.
Is it right? Furthemore, those loop in the literature are called the inner loop and outer loop and aren't nested, namely before I do the inner loop (the point 2 above) and later apart I do the outer loop i.e. a single cross validation on model with best parameters found in the inner loop. Is it right? How do you etablish the initial size of training set,test set and validation set?
Thanks in advance to who will help me. (I'm using svm, but this isn't very important)