I have built what I think is a very good predictive model using randomforest. The initial dataset was imbalanced for the outcome 2:1, so I randomly resampled the dataset to balance it, then trimmed the predictors down to 20 or so and managed to get the sensitivity and specificity of the model up to the 90s based on 10-fold cross validation.
Can I report that? Do I not have to test it on the imbalanced original dataset? I'm kind of afraid of these results as they look a bit too good, although they did take some man-hours to tune the machine to within an inch of its life. I've seen such things reported in the biomedical literature without a separately sourced validation dataset. Is 10-fold cross-validation "enough"? Essentially I want to make sure I haven't "cheated". Have I inflated the precision by resampling and if so what should I do about it? See below the WEKA buffer output.
=== Classifier model (full training set) === Random forest of 200 trees, each constructed while considering 5 random features. Out of bag error: 0.0199 Time taken to build model: 0.24 seconds === Stratified cross-validation === === Summary === Correctly Classified Instances 147 97.351 % Incorrectly Classified Instances 4 2.649 % Kappa statistic 0.947 Mean absolute error 0.0531 Root mean squared error 0.1419 Relative absolute error 10.6329 % Root relative squared error 28.384 % Coverage of cases (0.95 level) 100 % Mean rel. region size (0.95 level) 59.2715 % Total Number of Instances 151 === Detailed Accuracy By Class === TP Rate FP Rate Precision Recall F-Measure ROC Area Class 0.973 0.026 0.973 0.973 0.973 0.998 FALSE 0.974 0.027 0.974 0.974 0.974 0.998 TRUE Weighted Avg. 0.974 0.027 0.974 0.974 0.974 0.998 === Confusion Matrix === a | b <-- classified as 71 | 2 | a = FALSE 2 | 76 | b = TRUE