1
$\begingroup$

I was wondering if anyone might be able to provide some insight regarding how to interpret the results of some random forest hyperparameter tuning I am performing.

The training set consists of:

  • 1000 continuous features
  • 44 observations

The response I am trying to predict is continuous by nature, however, the distribution is bimodal so I am exploring both regression and classification.

Results from caret::train grid-based parameter tuning in R:

1. Regression:

Random Forest 

  44 samples
1000 predictors

No pre-processing
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 40, 39, 40, 40, 40, 40, ... 
Resampling results across tuning parameters:

  mtry  RMSE         Rsquared   MAE        
   2    0.001762244  0.8374687  0.001510297                                                               
   3    0.001763794  0.8220995  0.001507957
   4    0.001784599  0.8018954  0.001523252                                                               
   5    0.001785400  0.7992725  0.001528275                                                               
  10    0.001813925  0.7805094  0.001548873
  25    0.001862114  0.7484289  0.001588793
  50    0.001892789  0.7324827  0.001614362

RMSE was used to select the optimal model using the smallest value.
The final value used for the model was mtry = 2.

2. Classification:

Random Forest                                                                                             

  44 samples                                                                                              
1000 predictors                                                                                           
   2 classes: 'a', 'b'                                                                                    

No pre-processing
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 39, 39, 40, 40, 40, 40, ...                       
Resampling results across tuning parameters:       

  mtry  Accuracy  Kappa                                                    
   2    0.960     0.9090909
   3    0.960     0.9090909                      
   4    0.960     0.9090909
   5    0.960     0.9090909                                                                      
  10    0.960     0.9090909
  25    0.935     0.8590909 
  50    0.915     0.8136364                                                                     

Accuracy was used to select the optimal model using the largest value.         
The final value used for the model was mtry = 2. 

Prior feature selection

Prior to building this model, the features selected for inclusion in the training set were selected from a much larger set of possible features (the approach used for feature selection may warrant a separate discussion, but briefly, random subsets ~300 of all possible features were selected, highly correlated features were removed, and an RF model was trained using default mtry values on the random subset of features, and permutation-based variable importance scores were recorded. This process was repeated a large number of times and the variables with the highest average or max variable importance across all models were selected.)

Paramter tuning

Next, the caret package was used to try and select an optimal mtry value (similar to max_features in sklearn.ensemble.RandomForestClassifier) using 10-fold cross-validation:

# values of mtry to test
tune_grid <- data.frame(mtry=c(2, 3, 4, 5, 10, 25, 50))

# 10-fold cross validation (classProb set to TRUE for classification models)
train_control <- trainControl(method="cv", number=10, savePred=TRUE, classProb=FALSE)

train(response ~ ., data=training_set, method="rf", ntree=5000,
                    trControl=train_control, tuneGrid=tune_grid) 

In each case, across a number of different but related datasets, the optimal number of features to test at each split always appears to be at or near 2 (see example outputs above).

So my question is:

What conditions are likely to generate such low optimal mtry values?

One possibility that comes to mind is that the feature selection step resulted in a training set with many highly correlated features.

While there do indeed appear to be a decent amount of correlation structure in the training set, I'm not sure if it is sufficient to explain the low optimal mtry scores:

> cor_mat <- cor(training_set[, -1001], method='spearman')                                              
> quantile(cor_mat, probs=seq(0, 1, by=0.1))                                                              
         0%         10%         20%         30%         40%         50%                                   
-0.90500352 -0.32445384 -0.23044397 -0.15052854 -0.07019027  0.01451727                                   
        60%         70%         80%         90%        100%                                               
 0.09880545  0.17688513  0.25567301  0.35334743  1.00000000 

Any ideas?

Any insights or suggestions would be greatly appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.