3
$\begingroup$

I am trying to train my model using Scikit-learn's Random forest (Regression) and have tried to use GridSearch with Cross-validation (CV=5) to tune hyperparameters. I fixed n_estimators =2000 for all cases. Below are the few searches that I performed.

  1. max_features :[1,3,5], max_depth :[1,5,10,15], min_samples_split:[2,6,8,10], bootstrap:[True, False]

The best were max_features=5, max_depth = 15, min_samples_split:10, bootstrap=True

Best score = 0.8724

Then I searched close to the parameters that were best;

  1. max_features :[3,5,6], max_depth :[10,20,30,40], min_samples_split:[8,16,20,24], bootstrap:[True, False]

The best were max_features=5, max_depth = 30, min_samples_split:20, bootstrap=True

Best score = 0.8722

Again, I searched close to the parameters that were best;

  1. max_features :[2,4,6], max_depth :[25,35,40,50], min_samples_split:[22,28,34,40], bootstrap:[True, False]

The best were max_features=4, max_depth = 25, min_samples_split:22, bootstrap=True

Best score = 0.8725

Then I used GridSearch among the best parameters found in the above runs and found the best on as

max_features=4, max_depth = 15, min_samples_split:10, 

Best score = 0.8729

Then I used these parameters to predict for an unknown dataset but got a very low score (around 0.72).

My questions are;

  • I doing the hyperparameter tuning correctly or I am missing something?

  • Why is my testing score very low as compared to my training and validation score and how can I improve it so that I get good predictions out of my model?

Sorry, if these are basic questions as I am new to scikit-learn and ML.

P.S: The training (+Cross validation data) has 26138 samples with 6 features/inputs (columns) and one output. The testing data has 1416 samples.

$\endgroup$
  • 1
    $\begingroup$ You might be a little too tuned into your training set. This won't fix your CV error but try to do a 2/3rd training set and 1/3rd test set and see if that improves your test error. $\endgroup$ – Kristofersen Feb 5 '16 at 16:29
  • $\begingroup$ @Kristofersen: You mean I should not do Cross-validation while tuning hyperparameters, and use 2/3rd data for simple training and 1/3rd for testing? $\endgroup$ – Muhammad Feb 5 '16 at 16:40
  • $\begingroup$ I'm certainly no expert in this area, but whenever I've run random forests it's always been a 2/3rd 1/3rd split and run CV on the training set still. Also, isn't there some preprocessing step or postprocessing step that will clean up the random forrests? Why not just use the default settings? $\endgroup$ – Kristofersen Feb 5 '16 at 17:12
  • $\begingroup$ What you mean by clean up procedure? I think the settings will be different for different problems so that's why I am tuning hyperparameters. $\endgroup$ – Muhammad Feb 5 '16 at 17:34
  • $\begingroup$ The algorithm should be able to automatically determine the number of parameters / depth of the tree. In post processing it will build the full tree and then start trimming nodes at the bottom that don't contribute much to the prediction power. Try to run everything with default settings with 2/3 in training and see what the accuracy looks like $\endgroup$ – Kristofersen Feb 5 '16 at 17:36
3
$\begingroup$

I guess you have tunneled in on tuning too many non-useful hyper parameters, because an easy to use grid-search functionality allowed you to do so.

Notice all your explained variances only differ on the fourth digit. You have found, what appears to be a negligible better model settings. But even that you cannot be sure off because:

  • the RF model is non-deterministic and performance will vary slightly
  • a CV only estimates future model performance with a limited precision
  • nfold CV is not perfect reproducible and should be repeated to increase precision
  • Grid tuning should be performed with nested CV, but that is not your problem here I think.

Only "grid-tune" max_features. It has only 6 possoble values. You can run each 5 times and plot it. Check if some setting is repetitively better, probably you find anything from 2-4 perform fine. Max_depth is by default unlimited and that is optimal as long data is not very noisy. You set it to 25, which in practice is unlimited because already $2^{15}$=32000 and you "only" have 26000 samples. Changing these other hyper parameter will only give you shorter training times(useful) and/or more robust models. Thumb-rule: as explained variance is way above 50%, you do not need to make your model more robust by limiting depth of trees (max_depth, min_samples_split) to e.g. 3. Max_depth 15 is quite deep, and probably plenty deep enough, just as 2000 are trees enough. So raising and lowering number of trees and depth within the quite fine range does not change anything, and it will be really hard and non-rewarding to find the true best setting.

So you have performed a grid search and learned that RF will have the same performance in the parameter space you have tested.

If you obtain a testset from a different source you should expect a drop in performance. Your CV only estimate the model performance, if the future test set was drawn from the exactly same population. With 1400 tests, the sample error alone could swing the measured performance +/- 0.03, I guess.

If your swapped e.g. to boosting algorithms grid-tuning of multiple parameters would be a more rewarding tool.

To improve your model maybe you can refine your features. Look to variable importance, to see what features work well. Could you maybe derive new features with an even higher variable importance? Since your explained variance is quite high(low noise), you may benefit from swapping to xgboost. You may also spend time wondering if this chase of a better model performance of some target by some metric (explained variance) is useful specifically for your purpose. Maybe you don't need the model being that accurate when predicting large values, so you log transpose your target e.g. Maybe you only want to rank your predictions so explained variance could be replace with Spearman rank coefficient.

happy modelling:)

$\endgroup$
  • $\begingroup$ Thanks for your detailed answer. I have tried to look at the feature importance and found that one of the variables is not that important. I removed it and trained it again but did not give me better results. I have also tried to use 9 years climatic conditions worth of data (previously it was just 3 years) and used a high values of max_depth =100 and it gave me perfectly fit model (both on training and testing sets). But using that much data worries me that the testing data may already exist in those 9 years data (even if it is outside the training data). What you suggest? $\endgroup$ – Muhammad Feb 8 '16 at 11:37
  • $\begingroup$ In regards of very optimistic RF when trees a fully grown, it may be worth to check this question + 2 answers: stats.stackexchange.com/questions/66543/… $\endgroup$ – Soren Havelund Welling Feb 8 '16 at 16:03
  • $\begingroup$ dropping a single poor variable will not change much. Dropping 80 poor out of 100 variables may help. Tell me how it goes :) $\endgroup$ – Soren Havelund Welling Feb 8 '16 at 16:07
  • $\begingroup$ yeah using very distant in past time series etc. would be a problem if you expect the underlying system is an a transition these years. It's not easy to make a clear cut decision of how old data to use. Back testing may be the least poor option to know $\endgroup$ – Soren Havelund Welling Feb 8 '16 at 16:11
  • $\begingroup$ Well, there are only 6 variables (features) so I could drop one, which was less importance but could give a try by dropping another one. $\endgroup$ – Muhammad Feb 8 '16 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.