I have fit a random forest regression model to training data (used 65% of data for training). The data has approximately 40,000 observations and 100 features.
I fit a random forest regression in R with the following parameterization:
randomForest(formula = Response ~ .,
data = crs$dataset[, c(crs$input, crs$target)],
ntree = 500, mtry = 32, importance = TRUE, replace = FALSE, na.action = na.roughfix)
My understanding is that for Random Forest Regression problems, it is best to use approximately 1/3 of the candidate variables for each tree (rather than square root for classification problems) so that is why I have tried 32 variables per tree.
After applying the model to my test holdout data set (approximately 35% of data) the model appears to be overfit which I am confused by as I thought Random Forests were supposed to be rather resistant to overfit (which has been my experience in prior usage of them).
Here is a comparison of the average predicted vs. average actual value on Test data sorted ascending by predicted value (predictions grouped into deciles).
Prediction_Decile Avg_Prediction Avg_Actual Ratio:Actual/Predicted
1 4,570 6,343 1.388
2 5,939 7,085 1.193
3 6,789 7,429 1.094
4 7,576 7,982 1.054
5 8,320 8,981 1.079
6 9,105 8,796 0.966
7 9,954 8,657 0.870
8 10,977 9,306 0.848
9 12,304 9,814 0.798
10 14,653 10,195 0.696
As you can see the ratio of Actual to predicted value is steadily decreasing as the predictions increase which is why I think I am overfitting.
Any tips or advice on what may be causing this or how to tune the model to avoid this problem? The model appears to be doing a decent job of ordering the test observations,but a much poorer job of fitting them.