5
$\begingroup$

I would like to know how you would set the parameters for the GBM model?

1) I could optimize the parameters sequentially. At first, using a large value for shrinkage and a small number of iterations (for faster computation), I would try to optimize n.minobsinnode, interaction.depth, bag.fraction, etc. Then using these parameters, I would scale the skrinkage to lower value and find the optimal number of trees.

2) All at the same time, using tons of scenario simulations.
For example, is it possible that the optimal interaction.depth to have different values for shrinkage 0.01 and 0.001?
If yes, it would not be fine to fix some parameters to optimize the others...

$\endgroup$
4
$\begingroup$

In theory small shrinkage is supposed to always give a better result, but at the expense of more iterations required. When I have used GBM I have assumed this is true (admittedly without rigorously testing).

However, I wouldn't assume that the optimal set of interaction.depth, n.minobsinnode etc is the same for all values of the shrinkage (after optimising the number of iterations each time). Possibly this is true in some cases, and is probably roughly true in most. If you want to really squeeze the best performance possible without regard to computation cost I would not make this assumption at the outset.

$\endgroup$
  • $\begingroup$ Ok thank you. The issue is that optimizing the number of iterations is very long (especially with small shrinkage values). So finding optimal parameters all at the same time increases complexity exponentially. I wanted to find a better strategy. $\endgroup$ – Benoit_Plante Sep 10 '12 at 1:28
  • 1
    $\begingroup$ Yeah, it is long and painful to optimize GBM. If you do find a better strategy please share it! In practice I find there is a high variance in the optimal n.iterations which I circumvent somewhat by using Bagging, but it is very computationally expensive. $\endgroup$ – Bogdanovist Sep 10 '12 at 1:32
  • $\begingroup$ ok, then thank you. At least, I know now that there is no other way than the painful one. :) $\endgroup$ – Benoit_Plante Sep 11 '12 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.