Can we remove trees from a random forest with poor OOB error to improve generalisation? My objective is to improve out of sample generalization of my random forest while holding the number of trees constant. 
Suppose that I am only allowed to use $n$ trees on the out of sample data but it is computationally feasible to estimate $z$ trees on my in sample data where $z \geq n$ (from which, the best $n$ is selected from the $z$ estimated trees based on OOB error). Is there any research on what we expect the generalisation error to be as $z/n$ increases from $1$ to larger (theoretical asymptotic result or an actual empirical result)?
Will it offer better generalization than just choosing the trees randomly (equivalent to setting $n=z$)?
 A: Summary: I think in certain situations your strategy can help, but in others it won't and it may not be easy to know beforehand in which situation you are and it is going to be a small niche between bagging and boosting.

I think you are inventing something in the direction towards boosting here. Boosting weights the submodels according to their predictive performance (and then iteratively puts more weight on misclassified samples and so on). 
So I'd expect your strategy to be advantageous in situations where
- boosting starts to overfit, but
- bagging yields too low model complexity
- (In situations where boosting helps, your strategy probably helps to improve over bagging as well, but then you could use "normal" out-of-the-box boosting)
There are some points here you need to be aware of:


*

*As the oob performance is used to tune the model, you need an independent measurement of the generalization error - the internal oob estimate will be optimistically biased.
Any claims of superior boosting performance that was not validated with independent test data can be due to overfitting.

*As with any kind of tuning, you need to be careful about overfitting: 


*

*The more aggressive your tuning, the more prone you are to overfitting to noise on your data. 

*This also means: the more noise in your data, the more prone to overfitting you are with a given "aggressiveness" of tuning.
This is a known problem for boosting. 



Actually, the trees in the random forest are usually overfit, which is why the aggregation helps. So your strategy assumes implicitly that the random forest overcompensates the overfitting of the individual trees and you need a better fit to the data set. 
Also, you need to think about the misclassifications:


*

*I think you need to think carefully about what conclusions you can draw from the oob misclassifications of a single tree. Unless you have a large sample size, these estimates will be very uncertain (due to being tested with only ca. 1/3 of your total cases), and you'd want to avoid excluding trees that just accidentally appear to be bad because of the composition of the test data.

*Specifically, does the assumption that your data is (near to) perfectly separable (considering also the signal to noise ratio) make sense? 
