I am trying to build a binary classifier from a relatively large medical data set (1 - disease, 0 - no disease). The data set contains about 500 patients, of which 9% have the disease, and about 70 features (predictor variables). I would like to try:

  1. Feature selection - to reduce the number of features
  2. SMOTE for balancing the training dataset.
  3. Apply the classifier
  4. Apply cross-validation

Which is the best step by step approach?

Especially I have a dilemma about when to use feature selection? If I use it before SMOTE, the selected features might be biased? But applying it before, I have an unbalanced data problem.

Also, when is the right time to do cross-validation?

  • $\begingroup$ Which notion of classifier performance are you trying to optimize? $\endgroup$ – Kodiologist Jan 7 '18 at 16:24
  • $\begingroup$ Sensitivity, accuracy, precision, recall and F1 score $\endgroup$ – daz bones Jan 7 '18 at 17:39
  • $\begingroup$ Just a side thought - you could apply lasso regression first to see which features are less useful (they will go to zero), then remove those features and apply SMOTE / your desired classifier. You could also reduce the dimensionality of your features from 70 to K<70 using PCA. $\endgroup$ – PyRsquared Jan 7 '18 at 17:45
  • $\begingroup$ I was thinking about applying Lasso regression, but is this the correct way, applying it before SMOTE, due to the small proportion of diseased patients? $\endgroup$ – daz bones Jan 7 '18 at 17:59
  • $\begingroup$ 9% is not severely imbalanced, especially if the dataset is large. It is very doubtful if SMOTE is useful for your application. Why do you think you should use SMOTE or any other class balancing approach? $\endgroup$ – Matthew Drury Jan 7 '18 at 18:32

This paper argues that feature selection before SMOTE (Synthetic Minority Oversampling TEchnique) is preferred, and at a minimum:

... performing variable selection after using SMOTE should be done with some care because most variable selection methods assume that the samples are independent.

Oversampling the minority class with SMOTE violates the independence assumption.

For your application, it's not clear that SMOTE will provide any advantages over standard penalized approaches like LASSO or ridge regression. Also, be careful with your focus on "sensitivity, accuracy, precision, recall and F1 score" as optimization goals. They hide implicit assumptions about the relative costs of the two types of misclassifications. It's generally best to develop a reliable model for probabilities first, then (if classification is required) take the relative costs and benefits into account.

Cross-validation could certainly be used in your feature-selection process, for example choosing the penalty value for LASSO (and thus the number of features maintained). Note that the particular features selected by any algorithm are likely to differ from sample to sample, and you should consider that issue as you proceed.

Validating the entire process for developing the model, including the feature-selection process, could also be done with cross validation or, perhaps better, repeating the process on multiple bootstrap samples of the data.

  • $\begingroup$ Thank you for such a great answer. I will try it that way. It's generally best to develop a reliable model for probabilities first Do you have maybe a reference for this? Or an example to go through? $\endgroup$ – daz bones Jan 7 '18 at 18:57
  • $\begingroup$ This answer with its several useful links provides a good introduction to the issue. $\endgroup$ – EdM Jan 7 '18 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.