Random forest hyperparmeter tuning using Gradient descent I would like to use gradient descent for hyper parameter tuning for RF. Does any one has any insights on this, any merits and demerits would be appreciated? Also It would be great if some can share R code.
 A: To do gradient descent, you need continuous parameters, and the loss function has to be differentiable with respect to them. Random forests have discrete hyperparameters (e.g. tree depth, number of trees, number of features, etc.). So, unfortunately, gradient descent won't work in this context. Another point is that the loss function (presumably test set error) may have multiple local minima, and any local search procedure can get trapped.
Grid search and random search can be used to explore a broad range of hyperparameter space, and you can hone in on good regions after your initial search. Grid search is popular but, as Simone mentioned, random search can be faster in some cases. This is because some hyperparameters may not strongly affect performance relative to the others. Grid search will waste time exploring different values of the unimportant hyperparameters while holding the more important hyperparameters fixed. Random search updates all hyperparameters on each step, so it has a better chance of hitting the important ones.

Bergstra and Bengio (2012). Random Search for Hyper-Parameter Optimization.

A: I remember I attended an amazing presentation on hyperparameter tuning https://speakerdeck.com/tmls/keras-by-keisuke-kamataki-tmls-number-2
The speaker also pointed out how surprisingly random sampling of hyperparameters works very well.
