I have built what I think is a very good predictive model using randomforest. The initial dataset was imbalanced for the outcome 2:1, so I randomly resampled the dataset to balance it, then trimmed the predictors down to 20 or so and managed to get the sensitivity and specificity of the model up to the 90s based on 10-fold cross validation.

Can I report that? Do I not have to test it on the imbalanced original dataset? I'm kind of afraid of these results as they look a bit too good, although they did take some man-hours to tune the machine to within an inch of its life. I've seen such things reported in the biomedical literature without a separately sourced validation dataset. Is 10-fold cross-validation "enough"? Essentially I want to make sure I haven't "cheated". Have I inflated the precision by resampling and if so what should I do about it? See below the WEKA buffer output.

=== Classifier model (full training set) ===

Random forest of 200 trees, each constructed while considering 5 random features.
Out of bag error: 0.0199

Time taken to build model: 0.24 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances         147               97.351  %
Incorrectly Classified Instances         4                2.649  %
Kappa statistic                          0.947 
Mean absolute error                      0.0531
Root mean squared error                  0.1419
Relative absolute error                 10.6329 %
Root relative squared error             28.384  %
Coverage of cases (0.95 level)         100      %
Mean rel. region size (0.95 level)      59.2715 %
Total Number of Instances              151     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.973     0.026      0.973     0.973     0.973      0.998    FALSE
                 0.974     0.027      0.974     0.974     0.974      0.998    TRUE
Weighted Avg.    0.974     0.027      0.974     0.974     0.974      0.998

=== Confusion Matrix ===

  a   |  b   <-- classified as

 71    | 2 |  a = FALSE

  2    | 76 |  b = TRUE
  • $\begingroup$ When you say "I randomly resampled the dataset to balance it", does this relate to the stratified CV machinery (which is the default in Weka, as I seem to remember)? How were the 20 predictors selected? Did you work on the whole sample for building RF and assessing its performance, or did you keep an hold-out sample for validation apart? $\endgroup$ – chl Jan 9 '12 at 12:50
  • $\begingroup$ I used WEKA's facility for random resampling in the preprocessing stage which samples with replacement from both groups giving you a (fairly balanced) dataset. So from 151 cases it went from 100:51 to 78:73. I then developed a random Forest on that set and the results above are just 10-fold cross validation on that data set. I didn't interfere with the CV process after resampling the data. The 20 predictors were selected based on regression modelling and the VIMP of a random forest as well as domain knowledge. I didn't use a hold-out sample for the results above, although I have 99 case. $\endgroup$ – rosser Jan 9 '12 at 13:06
  • $\begingroup$ However, on a similarly unbalanced hold-out sample (this time 5:1 non-cases to cases) I am having the same difficulty as you would expect - the machine is very specific, but not very sensitive. $\endgroup$ – rosser Jan 9 '12 at 13:07
  • 1
    $\begingroup$ I would have thought that stratified sampling used to be applied directly when building the model, as discussed here: Using Random Forest to Learn Imbalanced Data (and on a related thread here). I would also consider a train/test sample (for optimizing RF parameters during CV) + another validation sample (to get a more reliable estimate of predictive accuracy). For selecting # predictors, you may rely on a permutation-based measure of variable importance. $\endgroup$ – chl Jan 9 '12 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.