Random Forest Regression Overfitting - Quantile Test on Test Data

Question

I have fit a random forest regression model to training data (used 65% of data for training). The data has approximately 40,000 observations and 100 features.

I fit a random forest regression in R with the following parameterization:

randomForest(formula = Response ~ .,
             data = crs$dataset[, c(crs$input, crs$target)],
             ntree = 500, mtry = 32, importance = TRUE, replace = FALSE, na.action = na.roughfix)

My understanding is that for Random Forest Regression problems, it is best to use approximately 1/3 of the candidate variables for each tree (rather than square root for classification problems) so that is why I have tried 32 variables per tree.

After applying the model to my test holdout data set (approximately 35% of data) the model appears to be overfit which I am confused by as I thought Random Forests were supposed to be rather resistant to overfit (which has been my experience in prior usage of them).

Here is a comparison of the average predicted vs. average actual value on Test data sorted ascending by predicted value (predictions grouped into deciles).

 Prediction_Decile  Avg_Prediction  Avg_Actual  Ratio:Actual/Predicted
 1  4,570   6,343   1.388 
 2  5,939   7,085   1.193 
 3  6,789   7,429   1.094 
 4  7,576   7,982   1.054 
 5  8,320   8,981   1.079 
 6  9,105   8,796   0.966 
 7  9,954   8,657   0.870   
 8  10,977  9,306   0.848 
 9  12,304  9,814   0.798  

10 14,653 10,195 0.696

As you can see the ratio of Actual to predicted value is steadily decreasing as the predictions increase which is why I think I am overfitting.

Any tips or advice on what may be causing this or how to tune the model to avoid this problem? The model appears to be doing a decent job of ordering the test observations,but a much poorer job of fitting them.

Answer 1

"nodesize: Minimum size of terminal nodes. Setting this number larger causes smaller trees to be grown (and thus take less time). Note that the default values are different for classification (1) and regression (5)."

Check out this parameter in the randomForest package. By upping the number of observations in terminal nodes you will have smaller (in terms of depth) trees. "Over-fit" trees are generally big (deep in terms of depth).

Answer 2

The Caret package enables you to run randomForests with a range of parameters for .mtry so you can optimise, using ROC values, the number of features that gives you optimal classification. This package also includes straightforward methods to carry out multiple iterations of k-fold cross validation to avoid overfitting.

I've included example code/syntax below.

set.seed(111)

#Set cross-validation parameters. The following code is for 3 repeats of 10-fold cross     
validation.


Features.CVparam <- trainControl(method = "repeatedcv", number = 10, repeats = 3, 
verboseIter = TRUE, returnData = TRUE, returnResamp = "all", classProbs = TRUE, 
summaryFunction = twoClassSummary)


#Create training grid for mtry values.
rf.traingrid <- expand.grid(.mtry = 35:45)

#Run RandomForests. Train.set is the training dataset, Train.class is a factor of class    
memberships, IIRC.

rf.model <- train(x = Train.set, y = Train.class, method = "rf", trControl =     
Features.CVparam,tuneGrid = rf.traingrid, metric = "ROC", varImp = TRUE, importance = 
TRUE, ntree = 500)

# print model parameters
print(rf.model)

# plot model parameters

plot(rf.model)

Additionally, the rfPermute package may be used to evaluate features that are significant contributors to classification efficiency.