10
$\begingroup$

I am trying to build a binary classifier from a relatively large medical data set (1 - disease, 0 - no disease). The data set contains about 500 patients, of which 9% have the disease, and about 70 features (predictor variables). I would like to try:

  1. Feature selection - to reduce the number of features
  2. SMOTE for balancing the training dataset.
  3. Apply the classifier
  4. Apply cross-validation

Which is the best step by step approach?

Especially I have a dilemma about when to use feature selection? If I use it before SMOTE, the selected features might be biased? But applying it before, I have an unbalanced data problem.

Also, when is the right time to do cross-validation?

Improve this question
$\endgroup$
8
  • $\begingroup$ Which notion of classifier performance are you trying to optimize? $\endgroup$
    – Kodiologist
    Commented Jan 7, 2018 at 16:24
  • $\begingroup$ Sensitivity, accuracy, precision, recall and F1 score $\endgroup$
    – daz bones
    Commented Jan 7, 2018 at 17:39
  • $\begingroup$ Just a side thought - you could apply lasso regression first to see which features are less useful (they will go to zero), then remove those features and apply SMOTE / your desired classifier. You could also reduce the dimensionality of your features from 70 to K<70 using PCA. $\endgroup$
    – PyRsquared
    Commented Jan 7, 2018 at 17:45
  • $\begingroup$ I was thinking about applying Lasso regression, but is this the correct way, applying it before SMOTE, due to the small proportion of diseased patients? $\endgroup$
    – daz bones
    Commented Jan 7, 2018 at 17:59
  • $\begingroup$ 9% is not severely imbalanced, especially if the dataset is large. It is very doubtful if SMOTE is useful for your application. Why do you think you should use SMOTE or any other class balancing approach? $\endgroup$
    – Matthew Drury
    Commented Jan 7, 2018 at 18:32

1 Answer 1

Reset to default
6
$\begingroup$

This paper argues that feature selection before SMOTE (Synthetic Minority Oversampling TEchnique) is preferred, and at a minimum:

... performing variable selection after using SMOTE should be done with some care because most variable selection methods assume that the samples are independent.

Oversampling the minority class with SMOTE violates the independence assumption.

For your application, it's not clear that SMOTE will provide any advantages over standard penalized approaches like LASSO or ridge regression. Also, be careful with your focus on "sensitivity, accuracy, precision, recall and F1 score" as optimization goals. They hide implicit assumptions about the relative costs of the two types of misclassifications. It's generally best to develop a reliable model for probabilities first, then (if classification is required) take the relative costs and benefits into account.

Cross-validation could certainly be used in your feature-selection process, for example choosing the penalty value for LASSO (and thus the number of features maintained). Note that the particular features selected by any algorithm are likely to differ from sample to sample, and you should consider that issue as you proceed.

Validating the entire process for developing the model, including the feature-selection process, could also be done with cross validation or, perhaps better, repeating the process on multiple bootstrap samples of the data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for such a great answer. I will try it that way. It's generally best to develop a reliable model for probabilities first Do you have maybe a reference for this? Or an example to go through? $\endgroup$
    – daz bones
    Commented Jan 7, 2018 at 18:57
  • $\begingroup$ This answer with its several useful links provides a good introduction to the issue. $\endgroup$
    – EdM
    Commented Jan 7, 2018 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.