0
$\begingroup$

I am training a random forest model with following parameters. n_estimators=50,criterion='mse',max_depth=None,min_samples_split=2, min_samples_leaf=1,min_weight_fraction_leaf=0.0,max_features='auto',max_leaf_nodes=None,oob_score=True My data set consists of 5258 rows and columns contain a categorical variable , 36 temperature data and 36 precipitation data. The model gives a R^2 score 0.81 on train data set where as only 0.35 on test data set. i have divided train and test data in 90:10 ratio.

Can i have suggestions how to improve the score?

$\endgroup$
  • $\begingroup$ What are you trying to predict with he random forest. $\endgroup$ – Michael Chernick Apr 12 '17 at 17:25
  • $\begingroup$ by definition you are overfitting to your training set. It is building a 'complicated' forest that is training itself to fit the noise in your training set as much as true signal. You want to constrain how complicated you let your model get, so that only the true signal shines through. It will lead to lower train scores, but higher test scores. A couple parameters to target - a low max_depth and a higher min_samples_split would be a good place to start. you also might consider using a tree-based library which has 'regularization' parameters, such as xgboost, and adjusting those. $\endgroup$ – Max Power Apr 12 '17 at 17:52
0
$\begingroup$

Following are some things that may work. I am assuming that you are using scikit-learn.

  1. use cross validation to avoid overfitting. It could be that the 10% test data that you use for validation is randomely the worst/noisy bit. k-fold cross validation would rotate the tezt subset k times to give you average scores with substantialy reduced overfitting. More information here: http://scikit-learn.org/stable/modules/cross_validation.html

  2. Other parameters can be optimized by testing range of values at once using Grid-search. More information here: http://scikit-learn.org/stable/modules/grid_search.html

Following are two parameters that often improve the performance of random forest. Note that they need more computational resources.

2.1 Number of trees (n_estimators) may be increased (500-1000) to increase the search space of the algorithm.

2.2 Nodes can be expanded till all leaves are pure (max_depth=None).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.