When performing cross-validation or bootstrap re sampling to estimate the performance of some machine learning algorithm, one commonly records the mean and variance of the errors obtained in of all the trials. This is commonly used in model selection, such as choosing the simplest (or quickest to run) model whose mean is within 1 sigma of the best mean (or some other rule of thumb).
Typically I have seen it stated that when using bootstrap aggregators (such as random forest) the out of bag error should be an unbiased estimate of the generalisation error, and therefore we don't need to bother doing cross-validation to estimate the error, we get that for free. In my experience this is usually true, however I don't see how I can get an estimate of the variance of that estimate, comparable to that which you get from CV or bootstrap resampling.
Is there a reasonable way to estimate this variance? We can't simply take the variance of the individual ensemble members (e.g. the trees in the case of random forest) since they will all have a much worse error than the ensemble (that being the point of bagging!). Is there some other reasonable approach?
Please note that statistical/machine learning is not my main field and I tend to get a bit muddled with the terminology at times. Please correct any misuse or confusing terminology.