3
$\begingroup$

I have been reading the questions related to nested cross validation and model selection and also gone through some tutorials. But I still do not understand how to solve the following problem: Suppose I have 2 classifiers: Logistic regression and Neural Network. I have some data (say 10000). I need to first find the best hyper-parameters for both of the classifiers and then also compare their performance. Here is what I think is reasonable:

  1. Create training set with 800 data points, keep 200 for testing
  2. Use k-fold cross validation with each of the classifiers to find the best hyper-parameters (such as regularizer or number of hidden nodes)
  3. Train both classifiers with total 800 data points and use the 200 data points for comparing the two classifiers.

I do not know if those steps are according to any standard procedure.

In several tutorials I found the process called nested CV, and here I get confused. If I use an outer loop for model comparison, and inner loop to select best parameter, then at each outer iteration, different hyper-parameter might be selected. But I want to find only one (the best) hyper parameter once and then compare the classifiers.

My questions are

  1. Are the previously mentioned steps follow any standard procedure?
  2. If not, how can I use Repeated/Nested CV in my case?
  3. I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that?

Thanks in advance.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ What you describe (k-fold CV on the training data and then evaluation of test set) if fine. If anything I would suggested repeated $k$-fold CV to get a bit more stable estimates. $\endgroup$ – usεr11852 Dec 7 '16 at 22:57
  • $\begingroup$ thanks @usεr11852. could you please refer some tutorials on repeated k-fold CV? $\endgroup$ – Rakib Dec 7 '16 at 23:04
  • $\begingroup$ Sorry no tutorials spring to mind. Molinaro's 2005 paper on : “Prediction Error Estimation: A Comparison of Resampling Methods.” is pretty straightforward I think though. You might want to also check: Kim's 2009: “Estimating Classification Error Rate: Repeated Cross-Validation, Repeated Hold-Out and Bootstrap.” This thread should be helpful too. $\endgroup$ – usεr11852 Dec 7 '16 at 23:20
  • 1
    $\begingroup$ @usεr11852 : a bit of an anti thesis but in addition to computation, repeated CV has other costs : On Estimating Model Accuracy with Repeated Cross-Validation (Vanwinckelen et. al. 2011). lirias.kuleuven.be/bitstream/123456789/346385/3/… $\endgroup$ – discipulus Dec 9 '16 at 1:22
  • $\begingroup$ @discipulus: +1. Thank you for the paper. I was unaware of it! Through a quick read the paper what I take back is that rep-CV is no panacea. Not that it is wrong and should not be done. It showcases that people should not (and indeed never should) treat rep-CV as a Holy Grail. (Also I do find the methodology of the paper a bit peculiar: Treating samples of less that 10k as a populations and then saying that rep-CV does not get the "true-population-value" seems a bit of stretch. Similarly folds of $N$ = 200 are quite small; their 1k folds fare obviously much better and are more realistic.) $\endgroup$ – usεr11852 Dec 9 '16 at 21:03
2
$\begingroup$

Are the previously mentioned steps follow any standard procedure? Yes! You are using hold-out validation set for final classifier comparison and k-fold cross-validation for the parameter (model) selection.

If not, how can I use Repeated/Nested CV in my case? Since you are considering different models, one way to improve that would be :

For each method

  1. Use k-fold cross-validation for model selection
  2. After selecting the optimal parameters (model fitting), use k-fold cross-validation to get the generalisation error.

This gives you the variation in errors in different folds, so you can calculate, variance (or standard deviation) to report on the reliability/consistency of the model, or even generate some plot.

UPDATE

You don't need to split the data for step 1 and step 2. Use 10000 data points in k-fold cross-validation, i.e., if k = 10, then you will use 9000 for training and 1000 for validation for model selection. Once model is selected again use the same 10000 samples in the similar k-fold cross-validation but this time your parameters will be fixed.

You can choose to run k-fold cross-validation once and get k error measures for each of the subset; 2*k if you consider training set which you could also look into. So, with those k or 2*k values you can perform some statistical tests or draw some plots. It is also good to repeat the cross-validation process n times, giving you n *k error measures for statistical analysis.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks. So your suggestion is (if I understood correctly), lets say I have 10000 data, then split it (what should be the ratio?) in two sub sets. use the first subset for cross-validation to select hyper parameters for each classifier. Then use the second subset for testing the final classifiers, but instead of using all at once, divide it further, test on each sub-subset and get average error and variance (how?) ? I also want to do statistical analysis on the accuracy of the 3 classifiers (e.g. t test), how should I do that? $\endgroup$ – Rakib Dec 8 '16 at 0:20
  • $\begingroup$ See my updated answer. $\endgroup$ – discipulus Dec 8 '16 at 1:16
  • $\begingroup$ If you do model fitting (hyper-parameter tuning) on the same data as the data used to estimate generalization error (for comparing two classifiers) then you will overfit to the data and your generalization error will be optimistic. Doing what Rakib suggested - splitting the data into two sub sets - is one way to avoid this. $\endgroup$ – MD004 Jan 29 '19 at 19:40
  • $\begingroup$ @MD004: Your point is valid because we have already used that data in hyper-parameter selection. If you have larger dataset, we can run cross-validation on independent sets. Nevertheless, we are using the process as independent processes. First identify hyper-parameters. Second, again retrain in cross-validation. For some cases when data is scarce, we are not leaking training and test data and also the assumption of any ML algorithm is that future data are similar to the current one but not necessarily the same. So, assumption could still hold. $\endgroup$ – discipulus Feb 2 '19 at 4:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.