Is it worth exploring OOB error to optimize random forest? The first time I used Random Forest, it gave me an accuracy of 81% with these parameters:
 n_estimators:10, max_features: sqrt(20), min_samples_split: 2 and min_samples_leaf: 1
Then I decided to optimize. 
1) I did a random search.
 Accuracy: 89.76%, n_estimators: 10, max_features:8, min_samples_split:6, min_samples_leaf:1. 
This took: 19 minutes.
2) Then I read that it is a good idea to uses OOB error to find optimal values for n_estimators and max_features. So I did it. The plot showed that with less than 100 trees the OOB error is high and inestable. And that max_features: 4 to 10 is good.
3) So I ran once more the random search. But I specified to look for trees over 100.
The best parameters are: n_ estimators: 127, max_features: 7, min_samples_split: 5 and min_samples_leaf: 1. 
Accuracy: 91,56%. Time: 6 hours!!!!!
As you can see the increase in accuracy is 3% more. But it took 6 hours.
So is it worth increasing the number of trees(n_estimators) from 10 to 127? 
 A: Breiman said in his original papers in Machine Learning journal, to not be "stingy" and use 5,000 to 50,000 trees -- but he had a lot of computational power.  Recall, that for RF, the in-bag bootstrapped objects are only used for training the trees, not testing.  Testing (i.e. error) is determined when the out-of-bag (OOB) objects are dropped down the trained trees, after which error is based on the percentage of class purity in the final nodes when node-splitting can no longer be performed.  Importance scores are also based on permuting feature-based columns in the OOB datasets, which gives loss of predictive accuracy as a function of each feature.   
I never use less than 1,000 trees for any RF runs.  Also, the default number of features randomly selected for node-splitting in a tree is typically srqt(total # of features).  You probably don't need to use the 6-hour run approach since you are hunting down better accuracy.  RF is one of the better classifiers, whose default parameter usage should do fine in terms of bias/variance and robustness across different data sets.  Recall, RFs generalize better than other classifiers, so the increased generalization comes at a cost of lower accuracy -- I am always comfortable with 75%-80% accuracy from RF, while other classifiers can be 90-95% accurate.  When you launch an ensemble of classifiers into the real world, you can always rest assured that the RF will probably do better than any of the other classifiers.  It won't get faked out, and probably won't break down, etc. 
