I am new to machine learning and have been trying to learn. I have a drug ingredients data as independent variables and its efficacy value as the dependent variable.

I split it into the train and test set (0.8:0.2) and fit the model to the training set. Then I tried to use the random forest method to do regression.

The resulting model when used to predict the train set give r2 accuracy of 0.97 and test set give 0.82. I have been trying to change the RF parameters but this is the highest test accuracy I could achieve.

I also find the OOB score of 0.85.

My question is, is this approach right? I have been researching about Randomforest in the internet and found the way Randomforest work is by doing bagging, therefore simulating CV. Is splitting the data the right thing to do? And can r2 score be used, or OOB score is the one to go to know the model accuracy? Maybe both?

Thank You.

  • 2
    $\begingroup$ A point to clarify terminology: your 'OOB score' is, I think, an OOB R^2 estimate. It might also refer to an OOB error rate, which is related but not the same thing. OOB estimates (either error rate or R^2) are very accurate while in-bag estimates are not, as the answers below state. $\endgroup$
    – mkt
    Jul 4 '17 at 9:32
  • $\begingroup$ Thx for the reply, it made me realize there are other terms for OOB. $\endgroup$
    – eljirg
    Jul 4 '17 at 10:05

In a cross-sectional data set (no time series or panel data), the OOB estimate of true performance of a random forest is usually very accurate and in my opinion can even replace (cross-)validation. Put differently, you can trust the OOB accuracy in such cases. This is in constrast to the insample (training set) accuracy: By construction, random forests tend to extremely overfit on the training data because the individual trees are usually very deep and unstable. So don't get lured by an insample accuracy/R-squared of 97%.

One warning: According to your description, you have used the test set to optimize hyperparameters of the model. This is inappropriate. The role of a test set is to get an impression of the performance of the final model. So you basically use it just once. There is no point of using the hold-out test sample in model optimization. It is very easy to overfit on the test data! That would be the role of a separate validation set, cross-validation or the OOB info.

  • $\begingroup$ Thank You for replying. The test set is used only to check the performance of the regression model that is trained on the training set. Is this not a valid step? $\endgroup$
    – eljirg
    Jul 4 '17 at 9:55
  • $\begingroup$ Ah I see your point after tinkering a bit, I will try the cross validation method too now. Thx! $\endgroup$
    – eljirg
    Jul 4 '17 at 10:02
  • $\begingroup$ Basically, the only decision you will do based on the test set is: "Will I be using the final model or not?" $\endgroup$
    – Michael M
    Jul 4 '17 at 11:11
  • $\begingroup$ You can certainly use the holdout sample for model tuning/validation purposes, but then you have to split your data three ways to get an honest estimate of the error: train, validate, and test. $\endgroup$
    – Hong Ooi
    Jul 4 '17 at 11:39

How you split your data is fine but it's not right to use your test case for improving your model.

... but this is the highest test accuracy ...

You can't do that. You should look at the cross-validation accuracy, that is your OOB.

It's fine to look at $R^2$, but OOB is generally considered the most unbiased approach. You should consider OOB to $R^2$ unless you have good reasons.

  • $\begingroup$ Thx for answering, I will try finding the best OOB thx! $\endgroup$
    – eljirg
    Jul 4 '17 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.