I'm training a Random Forest Regressor and I'm evaluating the performances. I have an MSE of 1116 on training and 7850 on the test set, suggesting me overfitting.

I would like to understand how to optimize the algorithm quality in generalization starting from cross-validation technique.

I did:

from sklearn.ensemble import RandomForestRegressor
from sklearn import model_selection
from sklearn import metrics
cv = model_selection.KFold(n_splits=8) 
for (train, test), i in zip(cv.split(X_train, y_train), range(8)):
    rfcv.fit(X_train.iloc[train], y_train.iloc[train])
    y_pred = rf.predict(X_test)
    print (metrics.mean_squared_error(y_test, y_pred))


accuracy = cross_val_score(model, df_final_X_hot, df_final_y, scoring='r2', cv = 10)

Now, I would like to understand how to use the results. what indication does cross validation give me? the algorithm that I have to use for my predictions is the one obtained from this cross validation?


Generally if your training error is much lower than the test error, it indeed suggests overfitting. However, the training error would almost always be smaller than the test error, and Random Forest generally doesn't overfit (as long as the bootstrap samples and mtry ratio are good- rule of thumb- 2/3, and sqrt(# variables) respectively). You could try different parameters for number of trees, bootstrap size and mtry as well as trees max depth (the deeper, the more prone to overfitting). Keep in mind that if anything, Random Forest tends to underfit (i.e too low variance in predictions)

Furthermore, because Random Forest uses bootstrap samples for training you don't need to perform cross validation as you get the 'out-of-bag error' - this is the error computed on the samples that were not selected in the bootstrap samples in each iteration. I am usually using R and not python so I can't tell you what is the name of the object but I'm sure it exists in sklearn.

  • $\begingroup$ @Ammon, yes it is in sklearn. so you are saying that if the oob score for the training is in line with the score for the test, it could not be overfitted. isn't it? $\endgroup$ Feb 4 at 13:46
  • 1
    $\begingroup$ Not really. The thing with the oob is that you get cv 'for free' without needing to sacrifice some of your training data for CV. just use the oob metric instead of performing cv. they should generally be in line (maybe you can check it for sanity check). In addition, Generally random forest doesn't overfit if parameters are used correctly (namely number of trees, bootstrap sample size, min/max tree depth, mtry). Still, if your training set is very small it might not fit well (hard to call it overfitting though) but in this case there is not much you can do- Maybe just look at the data manually $\endgroup$
    – Amnon
    Feb 4 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.