I have implemented a k-fold cross validation to to assess the classification performance of a Random Forest. What I want to know is: are the predicted values across folds directly comparable?
For example, when I generate predictions on holdout fold 1 and get a predicted value of 0.84 for one observation, can I be more confident in that prediction than a value of 0.80 for an observation in fold 2?
The ultimate question is if it would be appropriate to stack all of the predictions for my k-folds and then calculate model performance (such as ROC) from the stacked predictions. This could be useful in the case of highly imbalanced datasets with a low number of positives, as each fold will have an even lower number of positives and thus the ROC will have a relatively high variance across folds.
This post on RF was helpful, but does not directly address this question.
Addtional Info: I'm pariticularly interested in cases with high class imbalances and small positive sets. This doesn't change the question, but does highlight the potential issues with the comparing of results across folds.