2
$\begingroup$

I'm trying to tune the hyperparameters of my RandomForestRegressor created in python with sklearn with bootstrap = True using GridSearchCV. The hyperparameters that I am trying to tune are min_samples_leaf and min_samples_split (for the sole reason that those are the hyperparameters given by an experiment I'm trying to duplicate). Both these parameters concern the number of samples needed to either split or form a leaf.

Let's say I have 1000 samples and I use 750 for the training and 250 for the testing data (n_train = 750 and n_test = 250). Now I perform 3-fold cross-validation on my training data. There should be 500 samples available for training and 250 for validation in every iteration of the cross-validation. Does every tree in this case get 500 randomly bootstrapped samples or 750? If it is 500 I don't see the point in tuning min_samples_leaf and min_samples_split, because the number of samples in every tree in the grid search is different from the number of samples in a tree when training on the complete training data (There every tree should be build from 750 randomly selected samples with doubles, triples, etc. allowed). Or does GridSearchCV take this into account using the RandomForestRegressor by doing the grid search with the number of samples that would be used by a full training?

TLDR: Is a tree in a random forest regressor using GridSearchCV build from a number of samples equal to the size of the entire training data (n_train) or from a number of samples equal to the size of the subset of the training data used for training (in this case n_train/k * (k-1) = with k = 2 gives 500)?

$\endgroup$

2 Answers 2

0
$\begingroup$

U could try this. max_depth and n_estimators are the useful hyperparameters for RF regressor. min_samples_leaf and min_samples_split are not useful hyperparameters for RF regressor. Higher max_depth will contribute towards the overfitting.

seed(101)
from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search 
param_grid = {
    'max_depth': [2,3,4,5],
    'criterion' :['absolute_error','squared_error'],
    'n_estimators': [5,10,20,30, 40,50,60,70,80,100,150,200]
}
# Create a base model
rfCV = RandomForestRegressor(random_state = 1099)

# Instantiate the grid search model
regCV = GridSearchCV(estimator = rfCV, cv=5,param_grid = param_grid, n_jobs = -1, verbose = 2, return_train_score=True)
# Fit the grid search to the data
regCV.fit(X_train, y_train);
```
$\endgroup$
0
$\begingroup$

The trees are built with 500 examples in the search, then 750 examples for the refit model.

I don't see the point in tuning min_samples_leaf and min_samples_split, because the number of samples in every tree in the grid search is different from the number of samples in a tree when training on the complete training data

The two parameters min_samples_leaf and min_samples_split also accept float values in $(0,1]$, which are taken to mean the fraction of the training set size, which should alleviate your concern.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.