I was always taught 3 things:
Training algorithms (rf, trees, etc) don't perform well with unbalanced data.
I should balance data only after performing feature selection (mainly to keep variables independent)
Feature selection algorithms usually are based on training algorithms.
Taking these three points into consideration, how do I perform feature selection on an unbalanced data set?
After talking to many people, we all came to the conclusion that the best thing will be to separate the training and validation data and balance each separately. In this scenario, the feature selection will be done with synthetic data points, but they will belong only to the training set and won't "leak" to the validation/test set, thus I get the most objective feature selection possible in such a case.
Can anyone confirm this theory?