I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem: Suppose I have 2 classifiers: Logistic regression and Neural Network. I have some data (say 10000). I need to first find the best hyper-parameters for both of the classifiers and then also compare their performance. Here is what I think is reasonable:
- Create training set with 800 data points, keep 200 for testing
- Use k-fold cross validation with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes)
- Train both classifiers with total 800 data points and use the 200 data points for comparing the two classifiers.
I do not know if those steps are according to any standard procedure.
In several tutorials I found the process called nested CV, and here I get confused. If I use an outer loop for model comparison, and inner loop to select best parameter, then at each outer iteration, different hyper-parameter might be selected. But I want to find only one (the best) hyper parameter once and then compare the classifiers.
My questions are
- Are the previously mentioned steps follow any standard procedure?
- If not, how can I use Repeated/Nested CV in my case?
- I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that?
Thanks in advance.