1
$\begingroup$

So I am trying to implement Bayesian optimization for various machine learning methods, all of then consist of hyperparameters which should be tuned (eg. complexity parameter, minimum samples in split etc in decision tree, ....). According to help about function from R package rBayesianOptimization we have to specify

"FUN: The function to be maximized. This Function should return a named list with 2 components. The first component "Score" should be the metrics to be maximized, and the second component "Pred" should be the validation/cross-validation prediction for ensembling/stacking. "

So what will be the Score component, for example for this decision tree ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

The score should be whatever metric of model quality you want to maximize. Common choices for classification problems include accuracy, ROC AUC and log-likelihood.

Some metrics are conventionally minimized, such as cross-entropy, error rate, or Brier score. Of course you can just reverse the sign (multiply by $-1$) to transform the minimization into a maximization because a reversal of sign will only swap maxima and minima, but not change their location.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ +1. And in fear of stating the obvious, if we need to minimise a metric (e.g. Brier score) we can just take the said metric times $-1$ and then use this maximisation approach. The end results will be the "minimisation" of the original metric. $\endgroup$
    – usεr11852
    Dec 17, 2019 at 15:19
  • 1
    $\begingroup$ @usεr11852saysReinstateMonic I was afraid of over-complicating things by adding that detail but it seems that since both of us had the impulse, perhaps I should add it. Thanks for the suggestion. $\endgroup$
    – Sycorax ♦
    Dec 17, 2019 at 15:20
  • $\begingroup$ Is it better to use cross validation error or error on validation set for estimation of the generalization error? $\endgroup$
    – pikachu
    Dec 17, 2019 at 15:25
  • $\begingroup$ @pikachu Since this Answer seems to completely address your original Question, the question in your comment seems like a perfectly acceptable question to ask on its own. But before you do, please make a search of our archives to see if it's been asked and answered previously. $\endgroup$
    – Sycorax ♦
    Dec 17, 2019 at 15:36
  • $\begingroup$ Sorry for that. I have asked it already but I do not have answer so I wanted to seize the opportunity here. $\endgroup$
    – pikachu
    Dec 17, 2019 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.