0
$\begingroup$

I am using a random forest code to run one random forest model and distinguish which variables are important for classification and then to run a second random forest model using only these variables to reduce noise. The problem is that different variables are added to the model on different iterations of this process. I decided to repeat this process 10 times and use the model that produces the highest prediction accuracy (repeated measures sub-sampling from withheld data) as the model I report in my results. Is this a valid method?

$\endgroup$
2
  • $\begingroup$ Why do you think this would reduce noise? RF are designed to ignore noise on the fly. Unless you have many many variables and most of them are not informative you shouldnt worry about this. If you want to learn a bit more on variabel selection I recommend a video posted on youtube: Random Forests Theory and Applications for Variable Selection. $\endgroup$ – JEquihua Jul 12 '13 at 13:53
  • 1
    $\begingroup$ I've noticed that model prediction is quite a bit better if I don't add all variables. Here is a link to the code and references to papers where it's been used: evansmurphy.wix.com/evansspatial#!random-forests-model-select/… $\endgroup$ – Mina Jul 16 '13 at 21:20
2
$\begingroup$

Not really. Even though random forest promises to give you an OOB estimate of error rates, this is valid only for one run of the model. If you run the model repeatedly to get better results, you need a cross-validation strategy -- k-fold or LOO cross-validation -- to estimate your true error rates. Otherwise you are just throwing the dice as long as it takes to throw a six.

$\endgroup$
1
  • $\begingroup$ Thanks for your response. I am using a cross validation technique (repeated measures sub sampling) to estimate predictive accuracy rather than OOB estimates. I have to run 100 different models, so I wanted to make sure I was predicting using the best combination of variables. $\endgroup$ – Mina Jul 11 '13 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.