I have about 150 samples 1000 features (ranked by their importance by Relieff score). My question is, what would be the best approach to:

  • choose the hyper parameters

  • choose the optimal number of features to use

  • report the accuracy of my model using SVM and kNN (I don’t intend to choose which one of them is best to use, but rather report their accuracy)

First approach: Cross Validation

  1. Split data 80% training and 20% for final testing

  2. Using training data, perform feature ranking with Relieff score

  3. Using training data, loop over the K number of features (starting from the most to the least important) and hyper parameters, using 10-Fold cross validation (to computer the 10-Fold misclassification rate for each combination)

  4. Choose the best K (number of features) and Hyper parameters values, giving the least misclassification rate

  5. Train my algorithm using the training data and optimal parameters and test on the testing data (the 20% of my initial data, which were not used at all for selecting the parameters)

  6. Report accuracy

Second approach: Nested Cross Validation

  1. Split data into 10 folds (External Cross Validation)

  2. Do the same as above (Internal Cross Validation) to choose optimal K number of features, and hyper parameters using 10-fold cross validation.

  3. for each external fold, train using 9/10 of data with best chosen parameters and test using 1/10 of data

  4. report the average accuracy of the 10 external folds

Which one should I choose? Any suggestions?

  • $\begingroup$ This question might be relevant stats.stackexchange.com/questions/104713/…. Is Relieff score (assuming it is supervised) computed using the full dataset or just the training set? $\endgroup$ – user0 Jun 20 '17 at 13:39
  • $\begingroup$ Relieff score is computed using just the training set in both approaches $\endgroup$ – Learthgz Jun 20 '17 at 14:09

The first approach is actually hold out evaluation (although CV is used for tuning) and the second approach is cross validation IF you just consider the hyperparameters (eg, the feature importance and number of features and K, etc.) to be parameters of some modeling process that you intend to evaluate using cross validation. This is explained well in How to get hyper parameters in nested cross validation?.

If conceptualized this way, the answers in Hold-out validation vs. cross-validation become directly relevant. Some major benefits:

  • If you use hold out, you "lose" the testing data (in contrast, CV allows you to make statements about the generalization error of the model trained on the full dataset, so you don't waste any data). Sample size is a major consideration here, and I think with 150 observations the recommendation would be to use CV.

  • CV with its multiple folds gives a sense of the variability of the feature selection/hyperparameter optimization process as well as some measure of variability of performance. Clearly a modeling process with 0.90 accuracy $\pm$ 0.20 is not the same as 0.90 $\pm$ 0.02.

Another method that gives similar benefits is bootstrap: see Cross-validation or bootstrapping to evaluate classification performance?. This page also discusses that accuracy even without class imbalance is a poor scoring rule.

One difficulty with CV for modeling process evaluation (eg, nested CV) is that it requires that you, expectedly, automate your entire modeling process. So, anything that is subjective or manual is pretty much out of the question. Sometimes, domain expertise can only be integrated manually. Further, hyperparameter search must be automated, which is fairly easy, but so must be the search for the hyperparameter search space. For example, if you find in some fold F that your chosen K (for kNN) is actually at the border of your search space, you might want to expand the search space. If you don't do this, your comparison between kNN and SVM will not be valid because it's possibly that you gave SVM a better search space than you gave kNN. This search space expansion can only be done within fold F; there will be leakage if you have a globally defined search space used for all the folds that you change after seeing this (see Does changing the parameter search space after nested CV introduce optimistic bias?). This might take much longer to run (and be considerably more difficult to program) than a simple hold out.

  • $\begingroup$ Thank you very much for such detailed answer. So, if I choose the second approach (CV if I consider the hyperparameters to be parameters of some modeling process that you intend to evaluate), all I need to report at the end is the average accuracy of all folds? I don’t have to report the winning params of each fold? $\endgroup$ – Learthgz Jun 21 '17 at 1:58
  • $\begingroup$ Both average accuracy and winning parameters are useful for evaluating the classifier, although the accuracy estimate you get should be the expected estimate for that modeling process repeated on the full dataset (eg, stats.stackexchange.com/questions/282087/…). $\endgroup$ – user0 Jun 21 '17 at 14:00
  • $\begingroup$ The stability of the parameters is interesting (see stats.stackexchange.com/questions/65128/…). Eg, with 1000 features some might be colinear, causing some to be selected in some folds while others are selected in others. $\endgroup$ – user0 Jun 21 '17 at 14:00
  • $\begingroup$ Thank you so much! Before I accept your answer, just a last question about the variance and stability of my model. Using the 2nd approach (Nested CV) and optimizing K number of features and all SVM params (including Kernel function), the winning models of the 10 folds are (4 times SVM Linear, 4 times Polynomial Order 2 and 2 times Polynomial Order 3 ), but the accuracy of each fold is always between 92%-98%. Is my model stable in this case? $\endgroup$ – Learthgz Jun 22 '17 at 1:56
  • $\begingroup$ It's hard to say. Is it a binary classification task? If so, would you be able to plot the decision boundaries and see how much they change across models? Is the SVM regularized? Are the classes balanced? $\endgroup$ – user0 Jun 22 '17 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.