I was always taught 3 things:

  1. Training algorithms (rf, trees, etc) don't perform well with unbalanced data.

  2. I should balance data only after performing feature selection (mainly to keep variables independent)

  3. Feature selection algorithms usually are based on training algorithms.

Taking these three points into consideration, how do I perform feature selection on an unbalanced data set?


After talking to many people, we all came to the conclusion that the best thing will be to separate the training and validation data and balance each separately. In this scenario, the feature selection will be done with synthetic data points, but they will belong only to the training set and won't "leak" to the validation/test set, thus I get the most objective feature selection possible in such a case.

Can anyone confirm this theory?

  • 3
    $\begingroup$ Why do you need to select features at all? Feature selection has fewer benefits than people believe & a lot of potential pitfalls. $\endgroup$ Commented May 16, 2018 at 16:41
  • $\begingroup$ @gung can you point to some good sources arguin for not doing feature selection at all? $\endgroup$
    – Denwid
    Commented May 16, 2018 at 20:34
  • 6
    $\begingroup$ @Denwid, there aren't typically references for not doing things under any circumstances (except for the most common & egregious--eg median splits). That said, there are a ton of papers that have shown problems w/ the common methods; you can see some listed here: Backward selection for Cox model using R. It should be obvious that for any method you would it use only when it would be both valid & help you achieve your goals. The point is that lots of people believe they should select features (& thus do so) when those criteria aren't met. $\endgroup$ Commented May 16, 2018 at 20:57

3 Answers 3


In my experience feature selection tends to make performance worse rather than better if you are using a modern machine learning method that has some feature, such as regularisation, to avoid over-fitting. Miller's monograph on feature selection has similar advice hidden away in the appendices (sadly someone has borrowed my copy, so I can't find it). Basically feature selection is adding one binary degree of freedom to the learning problem for each input feature. This means the feature selection criteria can be reduced in two ways (i) getting rid of genuinely uninformative features (ii) selecting a set of features that happens to exploit some random sampling peculiarity of the data (i.e. overfitting). For an example of over-fitting in feature selection, see my answer to a related question about cross-validation and feature selection. The paper by Ambroise and MacLachlan is also well worth reading by anybody thinking of using feature selection with modern machine learning methods.

Secondly, the class imbalance problem is not really due to the imbalance itself, but because there are too few patterns belonging to the minority class to adequately describe it's distribution. Most classifiers work fine with imbalanced data provided you have a lot of data. Attempts to balance the dataset can make things worse rather than better by over-correcting for the bias due to class imbalance.

So if you have a performance problem due to class imbalance, it means you don't have enough data to adequately estimate the model parameters, in which case the last thing you should do is to perform feature selection as the added degrees of freedom this adds to the problem will only make the estimation problem worse. Regularisation is likely to be a much better solution as it adds essentially one continuous degree of freedom, and will be much less susceptible to over-fitting.


It seems that you are mixing two problems: 1) performing feature selection with an ensemble learning algorithm (e.g. random forest, RF); 2) balancing your dataset so the learning process of your algorithm is maximum.

For the first one, perhaps you could take a look to this paper, in which the authors propose a modification of RF (called Guided Regularized RF) to perform feature selection as well. There is an R implementation of this algorithm here that maybe is useful.

Then, the second problem is largely detached from the first one. In my experience, I have never seen a machine learning algorithm handling decently data imbalance by default (I am all ears if any reader has experienced the contrary), like a Poisson, Weibull or a Negative Binomial model would do. They are simply not fit for this task, at least in its basic form. But this does not have to be a problem: you can balance the classes yourself.

You should ensure an even number of samples belonging to each class during the training phase, and you should repeat this training phase using cross-validation techniques with random selection of samples to make sure that you are capturing most of the variance of the imbalanced class(es). In this way, the subsequent process of feature selection with RF should not be biased towards the imbalanced classes.

  • $\begingroup$ I also have the same doubts. $\endgroup$
    – Carlos AG
    Commented Dec 21, 2019 at 11:10
  • $\begingroup$ I have more than 60 features and my dataset is imbalanced 100:5 (class1:class2). Should I do feature selection first and then split train/test and balance the datasets? Or is it better to do the feature selection later? $\endgroup$
    – Carlos AG
    Commented Dec 21, 2019 at 11:12
  • $\begingroup$ @CarlosAG : Did you get any solution to this problem? $\endgroup$ Commented Oct 1, 2020 at 19:53
  • $\begingroup$ There are two ways: increasing the minority class (SMOTE technique) or decreasing the majority class (remove rows of that class at random). I did the second option, until the classes are 1:1. Also, I used cross validation of 10 iterations. $\endgroup$
    – Carlos AG
    Commented Oct 2, 2020 at 6:05

The question discusses more than one important topic. For the first one: There are many techniques to handle imbalance classes before learning a model or after the learning process. Techniques for balancing classes such as SMOTE and cost-sensitive learning and after learning a model, including the choice of performance measures that are less sensitive such as the AUC score. The second question, I'd suggest applying cross-validation then balancing the classes followed by the chosen feature selection technique. Regarding the third point, there are three categories of feature selection techniques (filters, wrappers and embedded techniques) and not all of them measure features' importance based on training a predictive model.

  • $\begingroup$ There is no way to want to balance a dataset without fundamentally misunderstanding the field of statistics. $\endgroup$ Commented Jul 14, 2021 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.