4
$\begingroup$

I have built what I think is a very good predictive model using randomforest. The initial dataset was imbalanced for the outcome 2:1, so I randomly resampled the dataset to balance it, then trimmed the predictors down to 20 or so and managed to get the sensitivity and specificity of the model up to the 90s based on 10-fold cross validation.

Can I report that? Do I not have to test it on the imbalanced original dataset? I'm kind of afraid of these results as they look a bit too good, although they did take some man-hours to tune the machine to within an inch of its life. I've seen such things reported in the biomedical literature without a separately sourced validation dataset. Is 10-fold cross-validation "enough"? Essentially I want to make sure I haven't "cheated". Have I inflated the precision by resampling and if so what should I do about it? See below the WEKA buffer output.

=== Classifier model (full training set) ===

Random forest of 200 trees, each constructed while considering 5 random features.
Out of bag error: 0.0199



Time taken to build model: 0.24 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances         147               97.351  %
Incorrectly Classified Instances         4                2.649  %
Kappa statistic                          0.947 
Mean absolute error                      0.0531
Root mean squared error                  0.1419
Relative absolute error                 10.6329 %
Root relative squared error             28.384  %
Coverage of cases (0.95 level)         100      %
Mean rel. region size (0.95 level)      59.2715 %
Total Number of Instances              151     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.973     0.026      0.973     0.973     0.973      0.998    FALSE
                 0.974     0.027      0.974     0.974     0.974      0.998    TRUE
Weighted Avg.    0.974     0.027      0.974     0.974     0.974      0.998

=== Confusion Matrix ===


  a   |  b   <-- classified as

 71    | 2 |  a = FALSE

  2    | 76 |  b = TRUE
Improve this question
$\endgroup$
5
  • $\begingroup$ When you say "I randomly resampled the dataset to balance it", does this relate to the stratified CV machinery (which is the default in Weka, as I seem to remember)? How were the 20 predictors selected? Did you work on the whole sample for building RF and assessing its performance, or did you keep an hold-out sample for validation apart? $\endgroup$
    – chl
    Commented Jan 9, 2012 at 12:50
  • $\begingroup$ I used WEKA's facility for random resampling in the preprocessing stage which samples with replacement from both groups giving you a (fairly balanced) dataset. So from 151 cases it went from 100:51 to 78:73. I then developed a random Forest on that set and the results above are just 10-fold cross validation on that data set. I didn't interfere with the CV process after resampling the data. The 20 predictors were selected based on regression modelling and the VIMP of a random forest as well as domain knowledge. I didn't use a hold-out sample for the results above, although I have 99 case. $\endgroup$
    – user6666
    Commented Jan 9, 2012 at 13:06
  • $\begingroup$ However, on a similarly unbalanced hold-out sample (this time 5:1 non-cases to cases) I am having the same difficulty as you would expect - the machine is very specific, but not very sensitive. $\endgroup$
    – user6666
    Commented Jan 9, 2012 at 13:07
  • 1
    $\begingroup$ I would have thought that stratified sampling used to be applied directly when building the model, as discussed here: Using Random Forest to Learn Imbalanced Data (and on a related thread here). I would also consider a train/test sample (for optimizing RF parameters during CV) + another validation sample (to get a more reliable estimate of predictive accuracy). For selecting # predictors, you may rely on a permutation-based measure of variable importance. $\endgroup$
    – chl
    Commented Jan 9, 2012 at 15:19
  • 1
    $\begingroup$ I’m voting to close this question because it is way too old. $\endgroup$
    – usεr11852
    Commented Apr 4, 2023 at 2:55

1 Answer 1

Reset to default
1
$\begingroup$

(If you’re afraid to evaluate your model on representative data, that should say a lot.)

By oversampling the minority class, you are telling the model to expect members is that category much more often than they truly appear. Consequently, if you go to representative data (natural class ratio), I suspect that you will find the model a bit trigger-happy to classify as being in the minority class, leading to more false positives than you might expect. Then if you do not care about false positives, save yourself the trouble of doing all this difficult machine learning work that risks melting your computer; just classify everything as being in that category. If this is unacceptable, then it would seem that you have some sense of the cost you incur from misclassifications, and you might be interested in a more nuanced assessment that uses the continuous outputs of your model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.