Comparing CV Predictions across Folds for Random Forest

Question

I have implemented a k-fold cross validation to to assess the classification performance of a Random Forest. What I want to know is: are the predicted values across folds directly comparable?

For example, when I generate predictions on holdout fold 1 and get a predicted value of 0.84 for one observation, can I be more confident in that prediction than a value of 0.80 for an observation in fold 2?

The ultimate question is if it would be appropriate to stack all of the predictions for my k-folds and then calculate model performance (such as ROC) from the stacked predictions. This could be useful in the case of highly imbalanced datasets with a low number of positives, as each fold will have an even lower number of positives and thus the ROC will have a relatively high variance across folds.

This post on RF was helpful, but does not directly address this question.

Addtional Info: I'm pariticularly interested in cases with high class imbalances and small positive sets. This doesn't change the question, but does highlight the potential issues with the comparing of results across folds.

Answer 1

For each fold, you are building a classifier that makes predictions for the observations. The classifiers within each fold have slightly different training sets and different weights, but they are all attempting to estimate the same underlying model. So yes, you can combine the predictions. If you have multiple predictions for one observation, you could take the average prediction of several folds, or weight the predictions so that the more accurate models have more influence than less accurate ones. This applies to any "ensemble learning" system. Predictions for different observations should be made on the same scale (e.g. from -1 to +1 or 0 to +1) so I can't think of any reason not to combine them.

Answer 2

After speaking with a few other folks about this problem, I think that technially you can't directly compare probabilities predicted for different folds, but practically, in most cases, you can.

The time when you would not be able to is if you have a small, potentially diverse positive set. Then when you divide the positives into k folds, each of the folds of positives may not be that similar to one another, so the k-1 folds are actually going to vary a bit; this would make the trees that compose each of the forests more different - this would seem to indicate that you couldn't directly compare the predicted probabilities across folds.

Now in practice, if you have a decently-sized positive set, then when you split those positives up across folds, each k-1 set of folds that composes the folds will be pretty similar, thus the forests will end up not being that different (assuming you have enough trees). So in practice the predicted probabilities will end up being close to directly comparable.

Answer 3

I am not sure you can combine all the predictions of the k-folds.

However, you could stratified your K folds so that you have similar number of positives in each fold and ROC performance would not vary due to imbalanced dataset.

In python, there is this package from scikit learn that works very well: http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedShuffleSplit.html#sklearn.cross_validation.StratifiedShuffleSplit

If you really have too few positive instances, you can work with bootstrapping instead of cross-validation (this paper explains it really well: http://scitation.aip.org/content/aapm/journal/medphys/35/4/10.1118/1.2868757 )