Feature Selection For Random Forest

Random Forest aims to combine many decision trees to make good predictions for testing data in regression and classification. It is an ensemble learning method.

I have a dataset with 100 samples, each of which has 400 features. I want to classify these samples. When I try to perform random forest classification, I get very low accuracy such as 0.53. According to some resources, there is no need of feature selection when applying random forest because it is very powerful method, and it chooses most important features. However, some people told me that you have many features; hence, at first you have to perform feature selection or pca before random forest classification.

I generally choose 10 maybe 20 features when performing random forest. It has a special parameter which specifies max features, and I choose 20 or 30 decision trees for classification. However, I cannot get a good accuracy.

What do you think about this feature selection issue ? Do we have to perform feature selection or pca before random forest ? Actually, I reduce the number of features during implementing random forest by using max_feature parameter, but it does not work.

rf_clf = RandomForestClassifier(criterion="entropy", n_estimators=20, max_features=10, n_jobs=2)
cv_kf = KFold(n_splits=5, shuffle=True, random_state=seed)

for train_index, test_index in cv_kf.split(features):

train_features = features[train_index]
train_labels = labels[train_index]
test_features = features[test_index]
test_labels = labels[test_index]

rf_clf.fit(train_features, train_labels)
predicted_labels = rf_clf.predict(test_features)
print(accuracy_score(test_labels, predicted_labels))

• "However, some people told me that you have many features; hence, at first you have to perform feature selection or pca before random forest classification." Why do you have to? What is that supposed to accomplish? It's true that RF doesn't inherently require feature selection when you have more features than samples. However, it's also true that no model does well when the sample size is small unless the signal is very strong. 20 trees is not many (try 100s) and all ML models need parameter tuning. max_features governs the number of features randomly selected at each split not total – Sycorax May 14 at 19:45
• Stated another way -- it seems "some people" are suggesting that the reason you have poor results is because RF is confused by having too many irrelevant features. You might have a better model if you take all irrelevant features out -- but that's true for any model. Finding only the relevant feature is challenging, so it's not unusual for people to let RF do its best at finding the good features. – Sycorax May 14 at 20:23
• I noticed that RF does not make pre-analysis to determine best relevant features. When it finishes classification, it determines the importance of features, but when performing classification it does not know their importance level. I performed Pearson Correlation Analysis to determine best features and reduced the dimension. Finally, I got good accuracy results. – Goktug May 15 at 17:34
• Random Forest isn’t unique in that — feature screening and model estimation are usually two distinct steps for ML models generally. – Sycorax May 15 at 18:01
• In a certain sense, RF is attempting to find the best features "along the way" by testing splits and then choosing which feature to split on. A feature with more information about the outcome will tend to be selected more often. This is what people mean when they say RF does its own feature selection. – Sycorax May 15 at 18:24