1
$\begingroup$

Does it make sense to use bayesian optimization for tuning of hyperparameters of decision tree model? I have not found any article or anything related to this, as BO is usually used for black-box algorithms.

$\endgroup$
  • 1
    $\begingroup$ Why wouldn't it make sense? $\endgroup$ – Tim Nov 8 at 17:02
  • $\begingroup$ to be honest I am not sure, BO was aimed to optimize black-box algorithms, DT is not black box bcs it is interpretable. However, I think the hyper-parameters still have to be tuned. But I have not found any mention about tuning of decision tree, as I have found for e.g. SVM, RF, NN, ... $\endgroup$ – pikachu Nov 8 at 17:29
3
$\begingroup$

Decision tree with thousands of variables and thousands of nodes would not really be interpretable. Very simple neural network will. There's no clear cut between black-box or not. Moreover it is an optimization algorithm, not an algorithm that works only with black-box functions.

$\endgroup$
  • $\begingroup$ Thanks! And do you think it make sense to prune the tree again after it was trained on optimized parameters? $\endgroup$ – pikachu Nov 14 at 13:24
3
$\begingroup$

It is perfectly fine to do so.

One might argue that optimising certain structural hyperparameters like the cost function (e.g. Gini index vs Entropy) is an extreme case of cherry-picking - we pick a model that works for a very specific snapshot of the data with total disregard to the problem mechanics. That assertion is probably correct but that said, using BO to optimise "standard" hyperparamters like the minimum number of items on a leaf or the maximum number of leaves, is totally reasonable.

$\endgroup$
  • $\begingroup$ Thanks! And do you think it make sense to prune the tree again after it was trained on optimized parameters? $\endgroup$ – pikachu Nov 14 at 13:25
  • 1
    $\begingroup$ Unfortunately, no. It does not; pruning should be part of the CV procedure as it is a hyperparameter step to optimise. $\endgroup$ – usεr11852 says Reinstate Monic Nov 14 at 15:10
  • $\begingroup$ So I can either use cross validation with bayesian optimization to choose the best complexity parameter and other parameters, OR train overfitted tree and afterwards prune it based on the lowest xerror ? $\endgroup$ – pikachu Nov 14 at 16:56
  • 1
    $\begingroup$ I would choose the first option. $\endgroup$ – usεr11852 says Reinstate Monic Nov 14 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.