Imbalanced data, SMOTE and feature selection I am trying to build a binary classifier from a relatively large medical data set (1 - disease, 0 - no disease). The data set contains about 500 patients, of which 9% have the disease, and about 70 features (predictor variables).
I would like to try:


*

*Feature selection - to reduce the number of features

*SMOTE for balancing the training dataset.

*Apply the classifier

*Apply cross-validation


Which is the best step by step approach?
Especially I have a dilemma about when to use feature selection? If I use it before SMOTE, the selected features might be biased? But applying it before, I have an unbalanced data problem.
Also, when is the right time to do cross-validation?
 A: This paper argues that feature selection before SMOTE (Synthetic Minority Oversampling TEchnique) is preferred, and at a minimum:

... performing variable selection after using SMOTE should be done with some care because most variable selection methods assume that the samples are independent.

Oversampling the minority class with SMOTE violates the independence assumption.
For your application, it's not clear that SMOTE will provide any advantages over standard penalized approaches like LASSO or ridge regression. Also, be careful with your focus on "sensitivity, accuracy, precision, recall and F1 score" as optimization goals. They hide implicit assumptions about the relative costs of the two types of misclassifications. It's generally best to develop a reliable model for probabilities first, then (if classification is required) take the relative costs and benefits into account.
Cross-validation could certainly be used in your feature-selection process, for example choosing the penalty value for LASSO (and thus the number of features maintained). Note that the particular features selected by any algorithm are likely to differ from sample to sample, and you should consider that issue as you proceed.
Validating the entire process for developing the model, including the feature-selection process, could also be done with cross validation or, perhaps better, repeating the process on multiple bootstrap samples of the data.
