I am using a random forest to solve a binary classification problem. Basically I am trying to maximize a score which depends on the values of TP, TN, FP and FN. FN has a more adverse effect on the score than FP. Also, the dataset is unbalanced. One class has a lot more instances than the other class. I am thinking of picking an optimal decision threshold on the probability to maximize the score. At the same time, I am trying to find an optimal selection of features to be fed into the random forest algorithm. I am using forward feature selection to solve this problem.

My question is, do picking the optimal decision threshold and picking the optimal set of feature interfere with each other? Should these two processes be done simultaneously? Or should one process be finished first and then proceed with the other one?

Thank you very much.

  • 2
    $\begingroup$ Random Forest is designed to be robust to unneeded features. My advice would be to just ignore the feature selection part unless absolutely necessary. $\endgroup$ – Matthew Drury Mar 29 '18 at 17:54
  • $\begingroup$ @matthew, thank you for your response. Ideally I would like to achieve both goals. Because calculation speed matters. The computation will take less time when the number of features are less. $\endgroup$ – wa1 Mar 29 '18 at 19:32
  • $\begingroup$ Then fit a single forest with all your features, remove the features it did not use, are refit on your reduced set of features. $\endgroup$ – Matthew Drury Mar 29 '18 at 19:34
  • $\begingroup$ @matthew, so you are saying that I can finish with the feature selection first, and then proceed with the optimization of the decision threshold? Thank you. $\endgroup$ – wa1 Mar 29 '18 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.