1
$\begingroup$

I am using LASSO and PCA for performing feature selection on a classification problem. The dataset consist of 20 features and around 5.7k observations. One of the reviewer comments for this approach is as follows:

Overfitting issue in feature selection

The authors reported using 3 feature selection techniques, namely Wilcoxon rank sum test, LASSO, and PCA, where only LASSO and PCA were used and evaluated. However, the feature selection in this study seems to be conducted with whole dataset but not within training dataset, which may result in overfitting. In fact, both feature selection and parameter tuning are recommended to be conducted in an independent dataset out from test dataset. Please refer to studies (Krstajic et al. Journal of Cheminformatics 2014, 6:10; Varma et al. BMC Bioinformatics. 2006; 7: 91.). A nested cross validation may provide a more unbiased result.

How will nested cross-validation help here? A 10-fold nested cross validation with feature selection could give me 10 possible set of selected features. In that case, how do we report the selected features? And also there could be different set of hyperparamters chosen for each fold as well. How do we report the optimal hyperparamters here? Could it be like feature selection could be done on the 50% of the data, rather than using the whole data.

|cite|improve this question
$\endgroup$
  • $\begingroup$ In this case, I don't think you need a technique sofisticated as a 10-fold nested CV. Maybe only separating your data in training and test datasets, with proportions like 70-30 or 80-20 can be enough. But as the reviewer recommended the nested CV, one approach would be to select the features that appear in most feature selections in the 10 ones you would perform in a 10-fold. You can then report only the most common features. As for the hyperparameters, I usually see people presenting their mean. $\endgroup$ – Bruna w Jun 23 '18 at 20:40
  • $\begingroup$ @Brunaw Thank you for the comment. Previously only two features were not selected, and when I do cross validation for feature section using LASSO, all the features seem relevant in at least 5 folds separately in 10-fold partitioning. In that case, I guess all features could be used. Is that right? And for PCA, 15 components sum to 99% variance in all 10 folds. And in the performance metrics table, which performance should I be putting, the one with nested cross validation or the one using the mean of hyperparameters? $\endgroup$ – prashanth Jun 23 '18 at 21:37
2
$\begingroup$

Selecting features and selecting hyper-parameters amount to the same thing. Your reviewer is correct that you should use nested CV to do this. You are also correct that each fold could produce a hyper-parameter/feature subset that is different from the others. The solution is NOT to choose the features that appear the most of the average value for the hyper-parameter as Bruna suggests, the solution is to use the nested CV process only to calculate an estimate of your generalization error. Once you have the generalization error from the nested CV process, take the whole data set and perform the feature selection / hyper-parameter selection gird search on the whole data set to find the best features/parameters, while keeping the error rate calculated from the nested CV.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ When you say "Once you have the generalization error from the nested CV process, take the whole data set and perform the feature selection / hyper-parameter selection..". What is the point of feature selection on full data and how can we relate it to the generalized error from nested CV. The readers might think that the selected features from the full data is used for final modelling which produced the generalized error. $\endgroup$ – prashanth Jun 24 '18 at 21:49
  • $\begingroup$ You have to understand the reason behind performing CV/nested CV. You don’t perform CV to build your model you perform CV to evaluate performance. You also have to realize that you aren’t evaluating a single models performance but rather a family of models and how it relates to the population of data that you have. You perform your grid search on full data once you have your estimate of generalization error because you want to build your model using as much data as possible to build the best model possible and in some cases help fight overfitting $\endgroup$ – astel Jun 25 '18 at 3:11
  • $\begingroup$ Okay. If I understand correctly, nested CV is used to report the generalized performance only and cannot be used to find final hyperparameter tuning or feature selection. To report the best hyperparamters for final model building and the features selected, full data can be used. Is that what you say? $\endgroup$ – prashanth Jun 25 '18 at 9:08
  • 1
    $\begingroup$ Yes this is what I am saying. You can read this question for perhaps a better explanation: stats.stackexchange.com/questions/65128/… $\endgroup$ – astel Jun 27 '18 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.