4
$\begingroup$

I am building my model in R and am using the randomForest package. My current model has 7 features and I see OOB error rate of about 14%. I also ran the rfcv in the random forest package to see how the error varies with the number of features. Here also I see an error rate of about 15% for 7 features.

However now when I apply this model to the test data my error rate blows up to 30%. Is this possible or is there an error in my code?

Train Set - 15,121 records
Test Set - 565,893 records
Both the data set were handed to me - I did not create them

Additional clarification and/or code available on request.

$\endgroup$
  • $\begingroup$ How did you create your training and test datasets? $\endgroup$ – Hong Ooi Oct 2 '14 at 3:44
  • $\begingroup$ What is your sample size in your train and test data? $\endgroup$ – Aghila Oct 2 '14 at 9:08
  • $\begingroup$ Lets me try to rephrase the question. I think one of reasons I am getting such poor results could be that the random forest is overfitting the data. (a)Is this possible (b) Is there an approach for detecting overfitting in random forests? $\endgroup$ – Abhi Oct 3 '14 at 17:24
  • $\begingroup$ It could be that you train does not contain a lot of the information that is in the test set. The oob error is computed using samples from the training set. This could explain the so different errors. Try to run random forest on train and test at the same time and check the oob error $\endgroup$ – Donbeo Nov 14 '14 at 23:28
  • $\begingroup$ Your test set is likely differently distributed to your training set. E.g. training on 2014 data testing on 2015 data when the distribution of the data is not constant between 2014 and 2015. $\endgroup$ – conjectures Jan 25 '16 at 13:09
3
$\begingroup$

The simple answer to your question is that your model is underfitted (i.e. not trained on enough instances) to the true distribution of the data. By definition, the out-of-bag estimate for a given model is as accurate as the model would be on a test set of the same size as the training set (Breiman,1996) . As your training set is comprised of 15,121 instances, your model should theoretically attain a testing error of 14-15% on a test set of 15,121 instances. However, as your model performs significantly worse on the significantly larger testing set, it suggests your model does not capture all the variations within the true distribution of the data. If you would like to improve your model's performance, consider training the model on more instances from the testing set, making sure to remove those instances from the testing set if you do. If this is not an option, consider performing boosting on your model to improve your generalization error.

Breiman, L. (1996). Out-of-bag estimation (pp. 1-13). Technical report, Statistics Department, University of California Berkeley, Berkeley CA 94708, 1996b. 33, 34.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.