Random Forest - What training set measure is the best predictor of test set accuracy? I'm running a random forest model on a training sample in R in order to make predictions on a hidden test set. I'm having difficulty in understanding how I should go about improving my model in order to make better predictions on the test set. There are a number of variables I can add to or remove from the model, but at the moment I am having to make decisions based on logic and gut feeling as opposed to using a formal method.


*

*As I understand it, the training set AUC for random forests is computed on the out-of-bag samples, so assuming that the nature of the data in the test set is similar to that of the training set, should I be looking to optimize this value?

*Does the out-of-bag error estimate have any bearing on how the model will perform on the test set? Does a lower value here indicate a better model that would be expected to perform better on the test set?
Any advice would be greatly appreciated!
 A: I'd recommend erring on the side of including more variables, since one of the selling points of random forests to begin with is that they can do a good job of using inputs variable according to how useful they actually are, heavily downweighting those which seem not to have much predictive value.
It looks like the AUC values R produces for random forests are using the whole training set, not just out-of-bag samples: http://r.789695.n4.nabble.com/Random-Forest-AUC-td3006649.html
Out-of-bag error is a lot like cross-validation error, so yes, it does estimate test error. That said, if you change the model or how it's fit after looking at out-of-bag error so as to minimize out-of-bag error, you may overfit for it and thus reduce performance given new test data. The same goes when doing cross-validation. The general principle is, the more aggressively you use some data for fitting, tuning, or selecting models, the more optimistically biased your estimates of test error will be using the same data, and the greater the danger of overfitting.
