Which hyperparameters should I choose to tune for different ML models? [closed]

Question

I'm applying hyperparameter tuning with techniques such as, randomized search and grid search in python. I could find important hyperparameters and tuned them for Random Forest Regressor algorithm, since there are many examples of this specific algorithm and I knew which hyperparameters are important to tune. But for other machine learning algorithms how should I choose important hyperparameters and define the space for them and tune them? I'm confused since each ML model has many hyperparameters and I know that only some of them are dominant and affect the result the most. In general, I'm wondering for different ML models which hyperparameters I should choose and tune.

Answer 1

For Random Forest Regressor the most important/useful parameter to tune would be n_estimators, this is how many decision trees are used in your model. Usually the default of 100 is pretty good, but it you want to try a few others you go try 10, 20, 50, 100, 200. If your dataset is really big you can try higher.

max_features is another good one to try. This is how many features to consider when making a split in the decision tree. You can try 0.25, 0.5, 0.75, 1.0 and the default of "sqrt".

You can try tune max_depth. This is the maximum depth of each decision tree. Be careful, as high numbers can cause you're model to overfit. If you think your model is overfitting reduce the max_depth, or vice versa if its underfitting you can increase max_depth.

If you're having trouble with overfitting increasing min_sample_leaf, which increases the minimum number of sample required in a leaf node, can help too

To answer your broader question, the best way to know what parameters to tune is to learn about how the model you're implementing works, and what each of the parameters does. Understanding the intricacies, strengths and weakness of different model architectures can be what separates a good data scientist from someone who throws spaghetti at the wall and sees what sticks.

In general I recommend against creating huge grid searches when hyper parameter tuning. This can often lead to overfitting. A step by step approach where you train the most influential parameter, followed by the next I find is often better. Of course there are parameters which are correlated and its good to grid search for them.

Some parameters are best left untouched unless you are having specific problems with the model eg. overfitting, underfitting and you can tune these parameters to remedy these issues.

And remember parameter tuning can't make a bad model good, it is only meant to squeeze performance (whether its accuracy or speed) out of your model