I am using a random forest code to run one random forest model and distinguish which variables are important for classification and then to run a second random forest model using only these variables to reduce noise. The problem is that different variables are added to the model on different iterations of this process. I decided to repeat this process 10 times and use the model that produces the highest prediction accuracy (repeated measures sub-sampling from withheld data) as the model I report in my results. Is this a valid method?
Not really. Even though random forest promises to give you an OOB estimate of error rates, this is valid only for one run of the model. If you run the model repeatedly to get better results, you need a cross-validation strategy -- k-fold or LOO cross-validation -- to estimate your true error rates. Otherwise you are just throwing the dice as long as it takes to throw a six.