Random Forest Regressor Python - cross validation

Question

I'm training a Random Forest Regressor and I'm evaluating the performances. I have an MSE of 1116 on training and 7850 on the test set, suggesting me overfitting.

I would like to understand how to optimize the algorithm quality in generalization starting from cross-validation technique.

I did:

from sklearn.ensemble import RandomForestRegressor
from sklearn import model_selection
from sklearn import metrics
rfcv=RandomForestRegressor()
cv = model_selection.KFold(n_splits=8) 
for (train, test), i in zip(cv.split(X_train, y_train), range(8)):
    rfcv.fit(X_train.iloc[train], y_train.iloc[train])
    y_pred = rf.predict(X_test)
    print (metrics.mean_squared_error(y_test, y_pred))


model=RandomForestRegressor()

accuracy = cross_val_score(model, df_final_X_hot, df_final_y, scoring='r2', cv = 10)
print(accuracy)

Now, I would like to understand how to use the results. what indication does cross validation give me? the algorithm that I have to use for my predictions is the one obtained from this cross validation?

Answer 1

Generally if your training error is much lower than the test error, it indeed suggests overfitting. However, the training error would almost always be smaller than the test error, and Random Forest generally doesn't overfit (as long as the bootstrap samples and mtry ratio are good- rule of thumb- 2/3, and sqrt(# variables) respectively). You could try different parameters for number of trees, bootstrap size and mtry as well as trees max depth (the deeper, the more prone to overfitting). Keep in mind that if anything, Random Forest tends to underfit (i.e too low variance in predictions)

Furthermore, because Random Forest uses bootstrap samples for training you don't need to perform cross validation as you get the 'out-of-bag error' - this is the error computed on the samples that were not selected in the bootstrap samples in each iteration. I am usually using R and not python so I can't tell you what is the name of the object but I'm sure it exists in sklearn.