0
$\begingroup$

So I am trying to implement Bayesian optimization for various machine learning methods, all of then consist of hyperparameters which should be tuned (eg. complexity parameter, minimum samples in split etc in decision tree, ....). According to help about function from R package rBayesianOptimization we have to specify

"FUN: The function to be maximized. This Function should return a named list with 2 components. The first component "Score" should be the metrics to be maximized, and the second component "Pred" should be the validation/cross-validation prediction for ensembling/stacking. "

So what will be the Score component, for example for this decision tree ?

$\endgroup$
3
$\begingroup$

The score should be whatever metric of model quality you want to maximize. Common choices for classification problems include accuracy, ROC AUC and log-likelihood.

Some metrics are conventionally minimized, such as cross-entropy, error rate, or Brier score. Of course you can just reverse the sign (multiply by $-1$) to transform the minimization into a maximization because a reversal of sign will only swap maxima and minima, but not change their location.

$\endgroup$
  • 2
    $\begingroup$ +1. And in fear of stating the obvious, if we need to minimise a metric (e.g. Brier score) we can just take the said metric times $-1$ and then use this maximisation approach. The end results will be the "minimisation" of the original metric. $\endgroup$ – usεr11852 Dec 17 '19 at 15:19
  • 1
    $\begingroup$ @usεr11852saysReinstateMonic I was afraid of over-complicating things by adding that detail but it seems that since both of us had the impulse, perhaps I should add it. Thanks for the suggestion. $\endgroup$ – Sycorax says Reinstate Monica Dec 17 '19 at 15:20
  • $\begingroup$ Is it better to use cross validation error or error on validation set for estimation of the generalization error? $\endgroup$ – pikachu Dec 17 '19 at 15:25
  • $\begingroup$ @pikachu Since this Answer seems to completely address your original Question, the question in your comment seems like a perfectly acceptable question to ask on its own. But before you do, please make a search of our archives to see if it's been asked and answered previously. $\endgroup$ – Sycorax says Reinstate Monica Dec 17 '19 at 15:36
  • $\begingroup$ Sorry for that. I have asked it already but I do not have answer so I wanted to seize the opportunity here. $\endgroup$ – pikachu Dec 17 '19 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.