I'm using the pretty great Bayesian Optimization package for python. I have a very noisy function I'd like to optimize for a given hyperparameter.

I've read a little on this, and it seems like if your objective function is really noisy, you want to use the alpha parameter. The creator of the package addressed this in a few issues:

https://github.com/fmfn/BayesianOptimization/issues/40 https://github.com/fmfn/BayesianOptimization/issues/115

He also linked to this scikit-learn page (that I believe he uses in his package), where they have the same alpha parameter. Their default value is 1e-10.

From messing around with it a little, I think I've found that a value of alpha=1 works for an example function I made (I say "think" because it seems to be pretty intermittent). Here is the example function, with the noiseless and noised version:

enter image description here

You can see that the noise version has a lot, but the ideal function does have a well enough defined maximum that it should be findable if it searches intelligently. And it tends to find it (repeating it for the same number of iterations):

enter image description here

it also sometimes still fails, like this:

enter image description here

But, this is a really simple contrived example function, with a known range and noise level. My real function, I'm not exactly sure what the range will be.

It feels like I've basically traded randomly searching for one hyperparameter with randomly searching for another.

If 1e-10 is "no noise" and 1.0 seems to work, that's...quite a range. Is there some strategy for figuring out what alpha value to choose?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.