I'm using the pretty great Bayesian Optimization package for python. I have a very noisy function I'd like to optimize for a given hyperparameter.
I've read a little on this, and it seems like if your objective function is really noisy, you want to use the alpha parameter. The creator of the package addressed this in a few issues:
He also linked to this scikit-learn page (that I believe he uses in his package), where they have the same alpha parameter. Their default value is
From messing around with it a little, I think I've found that a value of
alpha=1 works for an example function I made (I say "think" because it seems to be pretty intermittent). Here is the example function, with the noiseless and noised version:
You can see that the noise version has a lot, but the ideal function does have a well enough defined maximum that it should be findable if it searches intelligently. And it tends to find it (repeating it for the same number of iterations):
it also sometimes still fails, like this:
But, this is a really simple contrived example function, with a known range and noise level. My real function, I'm not exactly sure what the range will be.
It feels like I've basically traded randomly searching for one hyperparameter with randomly searching for another.
If 1e-10 is "no noise" and 1.0 seems to work, that's...quite a range. Is there some strategy for figuring out what alpha value to choose?