$R^2$ Score Vs OOB Score Random Forest

Question

I am new to machine learning and have been trying to learn. I have a drug ingredients data as independent variables and its efficacy value as the dependent variable.

I split it into the train and test set (0.8:0.2) and fit the model to the training set. Then I tried to use the random forest method to do regression.

The resulting model when used to predict the train set give $R^2$ accuracy of 0.97 and test set give 0.82. I have been trying to change the RF parameters but this is the highest test accuracy I could achieve.

I also find the OOB score of 0.85.

My question is, is this approach right? I have been researching about random forests on the internet and found the way random forests work is by doing bagging, therefore simulating CV. Is splitting the data the right thing to do? And can $R^2$ score be used, or OOB score is the one to go to know the model accuracy? Maybe both?

Answer 1

In a cross-sectional data set (no time series or panel data), the OOB estimate of true performance of a random forest is usually very accurate and in my opinion can even replace (cross-)validation. Put differently, you can trust the OOB accuracy in such cases. This is in constrast to the insample (training set) accuracy: By construction, random forests tend to extremely overfit on the training data because the individual trees are usually very deep and unstable. So don't get lured by an insample accuracy/R-squared of 97%.

One warning: According to your description, you have used the test set to optimize hyperparameters of the model. This is inappropriate. The role of a test set is to get an impression of the performance of the final model. So you basically use it just once. There is no point of using the hold-out test sample in model optimization. It is very easy to overfit on the test data! That would be the role of a separate validation set, cross-validation or the OOB info.

Answer 2

How you split your data is fine but it's not right to use your test case for improving your model.

... but this is the highest test accuracy ...

You can't do that. You should look at the cross-validation accuracy, that is your OOB.

It's fine to look at $R^2$, but OOB is generally considered the most unbiased approach. You should consider OOB to $R^2$ unless you have good reasons.