# Intuition behind low optimal # variables per split (mtry / max_features) during random forest tuning?

I was wondering if anyone might be able to provide some insight regarding how to interpret the results of some random forest hyperparameter tuning I am performing.

The training set consists of:

• 1000 continuous features
• 44 observations

The response I am trying to predict is continuous by nature, however, the distribution is bimodal so I am exploring both regression and classification.

Results from caret::train grid-based parameter tuning in R:

1. Regression:

Random Forest

44 samples
1000 predictors

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 40, 39, 40, 40, 40, 40, ...
Resampling results across tuning parameters:

mtry  RMSE         Rsquared   MAE
2    0.001762244  0.8374687  0.001510297
3    0.001763794  0.8220995  0.001507957
4    0.001784599  0.8018954  0.001523252
5    0.001785400  0.7992725  0.001528275
10    0.001813925  0.7805094  0.001548873
25    0.001862114  0.7484289  0.001588793
50    0.001892789  0.7324827  0.001614362

RMSE was used to select the optimal model using the smallest value.
The final value used for the model was mtry = 2.


2. Classification:

Random Forest

44 samples
1000 predictors
2 classes: 'a', 'b'

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 39, 39, 40, 40, 40, 40, ...
Resampling results across tuning parameters:

mtry  Accuracy  Kappa
2    0.960     0.9090909
3    0.960     0.9090909
4    0.960     0.9090909
5    0.960     0.9090909
10    0.960     0.9090909
25    0.935     0.8590909
50    0.915     0.8136364

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.


Prior feature selection

Prior to building this model, the features selected for inclusion in the training set were selected from a much larger set of possible features (the approach used for feature selection may warrant a separate discussion, but briefly, random subsets ~300 of all possible features were selected, highly correlated features were removed, and an RF model was trained using default mtry values on the random subset of features, and permutation-based variable importance scores were recorded. This process was repeated a large number of times and the variables with the highest average or max variable importance across all models were selected.)

Paramter tuning

Next, the caret package was used to try and select an optimal mtry value (similar to max_features in sklearn.ensemble.RandomForestClassifier) using 10-fold cross-validation:

# values of mtry to test
tune_grid <- data.frame(mtry=c(2, 3, 4, 5, 10, 25, 50))

# 10-fold cross validation (classProb set to TRUE for classification models)
train_control <- trainControl(method="cv", number=10, savePred=TRUE, classProb=FALSE)

train(response ~ ., data=training_set, method="rf", ntree=5000,
trControl=train_control, tuneGrid=tune_grid)


In each case, across a number of different but related datasets, the optimal number of features to test at each split always appears to be at or near 2 (see example outputs above).

So my question is:

What conditions are likely to generate such low optimal mtry values?

One possibility that comes to mind is that the feature selection step resulted in a training set with many highly correlated features.

While there do indeed appear to be a decent amount of correlation structure in the training set, I'm not sure if it is sufficient to explain the low optimal mtry scores:

> cor_mat <- cor(training_set[, -1001], method='spearman')
> quantile(cor_mat, probs=seq(0, 1, by=0.1))
0%         10%         20%         30%         40%         50%
-0.90500352 -0.32445384 -0.23044397 -0.15052854 -0.07019027  0.01451727
60%         70%         80%         90%        100%
0.09880545  0.17688513  0.25567301  0.35334743  1.00000000


Any ideas?

Any insights or suggestions would be greatly appreciated.