I was always taught 3 things:

  1. Training algorithms (rf, trees, etc) don't perform well with unbalanced data.

  2. I should balance data only after performing feature selection (mainly to keep variables independent)

  3. Feature selection algorithms usually are based on training algorithms.

Taking these three points into consideration, how do I perform feature selection on an unbalanced data set?


After talking to many people, we all came to the conclusion that the best thing will be to separate the training and validation data and balance each separately. In this scenario, the feature selection will be done with synthetic data points, but they will belong only to the training set and won't "leak" to the validation/test set, thus I get the most objective feature selection possible in such a case.

Can anyone confirm this theory?

  • 1
    $\begingroup$ Why do you need to select features at all? Feature selection has fewer benefits than people believe & a lot of potential pitfalls. $\endgroup$ – gung - Reinstate Monica May 16 '18 at 16:41
  • $\begingroup$ @gung can you point to some good sources arguin for not doing feature selection at all? $\endgroup$ – Denwid May 16 '18 at 20:34
  • 4
    $\begingroup$ @Denwid, there aren't typically references for not doing things under any circumstances (except for the most common & egregious--eg median splits). That said, there are a ton of papers that have shown problems w/ the common methods; you can see some listed here: Backward selection for Cox model using R. It should be obvious that for any method you would it use only when it would be both valid & help you achieve your goals. The point is that lots of people believe they should select features (& thus do so) when those criteria aren't met. $\endgroup$ – gung - Reinstate Monica May 16 '18 at 20:57

It seems that you are mixing two problems: 1) performing feature selection with an ensemble learning algorithm (e.g. random forest, RF); 2) balancing your dataset so the learning process of your algorithm is maximum.

For the first one, perhaps you could take a look to this paper, in which the authors propose a modification of RF (called Guided Regularized RF) to perform feature selection as well. There is an R implementation of this algorithm here that maybe is useful.

Then, the second problem is largely detached from the first one. In my experience, I have never seen a machine learning algorithm handling decently data imbalance by default (I am all ears if any reader has experienced the contrary), like a Poisson, Weibull or a Negative Binomial model would do. They are simply not fit for this task, at least in its basic form. But this does not have to be a problem: you can balance the classes yourself.

You should ensure an even number of samples belonging to each class during the training phase, and you should repeat this training phase using cross-validation techniques with random selection of samples to make sure that you are capturing most of the variance of the imbalanced class(es). In this way, the subsequent process of feature selection with RF should not be biased towards the imbalanced classes.


The question discusses more than one important topic. For the first one: There are many techniques to handle imbalance classes before learning a model or after the learning process. Techniques for balancing classes such as SMOTE and cost-sensitive learning and after learning a model, including the choice of performance measures that are less sensitive such as the AUC score. The second question, I'd suggest applying cross-validation then balancing the classes followed by the chosen feature selection technique. Regarding the third point, there are three categories of feature selection techniques (filters, wrappers and embedded techniques) and not all of them measure features' importance based on training a predictive model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.