I am using LASSO and PCA for performing feature selection on a classification problem. The dataset consist of 20 features and around 5.7k observations. One of the reviewer comments for this approach is as follows:
Overfitting issue in feature selection
The authors reported using 3 feature selection techniques, namely Wilcoxon rank sum test, LASSO, and PCA, where only LASSO and PCA were used and evaluated. However, the feature selection in this study seems to be conducted with whole dataset but not within training dataset, which may result in overfitting. In fact, both feature selection and parameter tuning are recommended to be conducted in an independent dataset out from test dataset. Please refer to studies (Krstajic et al. Journal of Cheminformatics 2014, 6:10; Varma et al. BMC Bioinformatics. 2006; 7: 91.). A nested cross validation may provide a more unbiased result.
How will nested cross-validation help here? A 10-fold nested cross validation with feature selection could give me 10 possible set of selected features. In that case, how do we report the selected features? And also there could be different set of hyperparamters chosen for each fold as well. How do we report the optimal hyperparamters here? Could it be like feature selection could be done on the 50% of the data, rather than using the whole data.