I am using a random forest to solve a binary classification problem. Basically I am trying to maximize a score which depends on the values of TP, TN, FP and FN. FN has a more adverse effect on the score than FP. Also, the dataset is unbalanced. One class has a lot more instances than the other class. I am thinking of picking an optimal decision threshold on the probability to maximize the score. At the same time, I am trying to find an optimal selection of features to be fed into the random forest algorithm. I am using forward feature selection to solve this problem.
My question is, do picking the optimal decision threshold and picking the optimal set of feature interfere with each other? Should these two processes be done simultaneously? Or should one process be finished first and then proceed with the other one?
Thank you very much.