# What do we want to maximize in hyperparameter tuning with bayesian optimization?

So I am trying to implement Bayesian optimization for various machine learning methods, all of then consist of hyperparameters which should be tuned (eg. complexity parameter, minimum samples in split etc in decision tree, ....). According to help about function from R package rBayesianOptimization we have to specify

"FUN: The function to be maximized. This Function should return a named list with 2 components. The first component "Score" should be the metrics to be maximized, and the second component "Pred" should be the validation/cross-validation prediction for ensembling/stacking. "

So what will be the Score component, for example for this decision tree ?

The score should be whatever metric of model quality you want to maximize. Common choices for classification problems include accuracy, ROC AUC and log-likelihood.

Some metrics are conventionally minimized, such as cross-entropy, error rate, or Brier score. Of course you can just reverse the sign (multiply by $$-1$$) to transform the minimization into a maximization because a reversal of sign will only swap maxima and minima, but not change their location.

• +1. And in fear of stating the obvious, if we need to minimise a metric (e.g. Brier score) we can just take the said metric times $-1$ and then use this maximisation approach. The end results will be the "minimisation" of the original metric. Dec 17, 2019 at 15:19
• @usεr11852saysReinstateMonic I was afraid of over-complicating things by adding that detail but it seems that since both of us had the impulse, perhaps I should add it. Thanks for the suggestion.
– Sycorax
Dec 17, 2019 at 15:20
• Is it better to use cross validation error or error on validation set for estimation of the generalization error? Dec 17, 2019 at 15:25
• @pikachu Since this Answer seems to completely address your original Question, the question in your comment seems like a perfectly acceptable question to ask on its own. But before you do, please make a search of our archives to see if it's been asked and answered previously.
– Sycorax
Dec 17, 2019 at 15:36
• Sorry for that. I have asked it already but I do not have answer so I wanted to seize the opportunity here. Dec 17, 2019 at 15:38