While studying hyperparameter tuning in Machine Learning, I have come to read Bayesian Optimization for Hyperparameter Tuning and Using validation set when training the model but it is kind of ambiguous to me to tell the usage of Bayesian Optimization and validation set when training the model. Using validation data set, I can measure the performance of the model adjusting the hyperparameters but I don't know when to use Bayesian Optimization for hyperparameter tuning. Hope to hear some explanations about it.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
Bayesian optimization (BO) is a recipe to tell how you should explore the hyper-parameter (HP) space. So, you'll be using your validation set as before, but exploring the space in the direction that BO suggests. This is typically useful when you have a lot of HPs to tune and it's computationally expensive (or impossible) to try them all. Grid search explores the HP space in an ordered manner; random search just randomly picks some HP values but BO selects the next HPs to try based on how much information it'd gain from trying that HPs with your model. It's much more economic for a large and high dimensional HP space.
