5
$\begingroup$

I have implemented a k-fold cross validation to to assess the classification performance of a Random Forest. What I want to know is: are the predicted values across folds directly comparable?

For example, when I generate predictions on holdout fold 1 and get a predicted value of 0.84 for one observation, can I be more confident in that prediction than a value of 0.80 for an observation in fold 2?

The ultimate question is if it would be appropriate to stack all of the predictions for my k-folds and then calculate model performance (such as ROC) from the stacked predictions. This could be useful in the case of highly imbalanced datasets with a low number of positives, as each fold will have an even lower number of positives and thus the ROC will have a relatively high variance across folds.

This post on RF was helpful, but does not directly address this question.

Addtional Info: I'm pariticularly interested in cases with high class imbalances and small positive sets. This doesn't change the question, but does highlight the potential issues with the comparing of results across folds.

$\endgroup$
1
+100
$\begingroup$

For each fold, you are building a classifier that makes predictions for the observations. The classifiers within each fold have slightly different training sets and different weights, but they are all attempting to estimate the same underlying model. So yes, you can combine the predictions. If you have multiple predictions for one observation, you could take the average prediction of several folds, or weight the predictions so that the more accurate models have more influence than less accurate ones. This applies to any "ensemble learning" system. Predictions for different observations should be made on the same scale (e.g. from -1 to +1 or 0 to +1) so I can't think of any reason not to combine them.

$\endgroup$
  • $\begingroup$ My concern with combining them is that I think the predicted probabilities would be based on the votes, and the votes will come from a different set of trees for each fold. So it would be close to being right, and is therefore fine in most situations, but I think it still remains that a .81 in one fold is not directly comparable to a .82 in another fold. $\endgroup$ – Tchotchke Aug 26 '15 at 12:56
1
$\begingroup$

After speaking with a few other folks about this problem, I think that technially you can't directly compare probabilities predicted for different folds, but practically, in most cases, you can.

The time when you would not be able to is if you have a small, potentially diverse positive set. Then when you divide the positives into k folds, each of the folds of positives may not be that similar to one another, so the k-1 folds are actually going to vary a bit; this would make the trees that compose each of the forests more different - this would seem to indicate that you couldn't directly compare the predicted probabilities across folds.

Now in practice, if you have a decently-sized positive set, then when you split those positives up across folds, each k-1 set of folds that composes the folds will be pretty similar, thus the forests will end up not being that different (assuming you have enough trees). So in practice the predicted probabilities will end up being close to directly comparable.

$\endgroup$
  • 3
    $\begingroup$ I think that if even if the data sets in different folds are very diverse, each tree is still gives an unbiased estimate of the class labels. The high variance means that you may be combining 'good' predictions with 'bad' predictions, but that's still legitimate. $\endgroup$ – dcorney Aug 27 '15 at 10:00
0
$\begingroup$

I am not sure you can combine all the predictions of the k-folds.

However, you could stratified your K folds so that you have similar number of positives in each fold and ROC performance would not vary due to imbalanced dataset.

In python, there is this package from scikit learn that works very well: http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedShuffleSplit.html#sklearn.cross_validation.StratifiedShuffleSplit

If you really have too few positive instances, you can work with bootstrapping instead of cross-validation (this paper explains it really well: http://scitation.aip.org/content/aapm/journal/medphys/35/4/10.1118/1.2868757 )

$\endgroup$
  • 1
    $\begingroup$ Thanks for your attention to this question, but I don't feel that this answer directly addresses OP's concern: is it possible to directly compare predicted values across folds? $\endgroup$ – Sycorax Aug 20 '15 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.