3
$\begingroup$

I am performing Bayesian Optimization to select a hyperparameter configuration for my supervised learning model. I understand that with each additional hyperparameter that I choose to optimize, the search space grows exponentially. My initial idea was to increase the number of search iterations exponentially as search space dimensionality increases, but I realized that it wouldn't offer much (if any) benefit over using a grid search, whose runtime would increase in the same way with added dimensions.

Is there some rule of thumb for increasing the number of iterations as search space dimensionality increases? For what it's worth, my BO algorithm uses a Gaussian process as the surrogate model, and I'm leaning towards using a linear function, specifically $n = 20*D$, where $D$ is the search space dimension.

Improve this question
$\endgroup$
3
  • $\begingroup$ A relevant benchmark is to consider whether your BO procedure can beat random search. Typically, random search is run for 60 iterations. stats.stackexchange.com/a/193310/22311 So to be favorable to random search, your BO procedure would need to achieve a better result in 60 iterations, or an equivalent result in less than 60 iterations. $\endgroup$
    – Sycorax ♦
    Commented Nov 13, 2023 at 22:15
  • $\begingroup$ @Sycorax My concern with using 60 iterations of a random search is that in high dimensional space, simply being in the top 5% still may not be very good. $\endgroup$
    – David
    Commented Nov 13, 2023 at 22:22
  • 1
    $\begingroup$ Of course, and that's a fair criticism of random search; I've pointed this out elsewhere stats.stackexchange.com/a/561195/22311 But what I'm saying is that you'll want to show that you're getting value for the effort of using BO -- showing that BO gets better faster than RS does. In your linear rule, if $D=3$, then you're doing $n=60$ queries of the hyperparameter space (training 60 models), just as random search, but you're also doing the BO procedure at each iteration (increasing computation cost). Moreover, if $D>3$, then you're fitting more models than RS typically does. $\endgroup$
    – Sycorax ♦
    Commented Nov 13, 2023 at 22:26

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.