How can I determine what values of alpha and kappa to use for Bayesian Optimization? I'm using the pretty great Bayesian Optimization package for python. I have a very noisy function I'd like to optimize for a given hyperparameter.
I've read a little on this, and it seems like if your objective function is really noisy, you want to use the alpha parameter. The creator of the package addressed this in a few issues:
https://github.com/fmfn/BayesianOptimization/issues/40
https://github.com/fmfn/BayesianOptimization/issues/115
He also linked to this scikit-learn page (that I believe he uses in his package), where they have the same alpha parameter. Their default value is 1e-10.
From messing around with it a little, I think I've found that a value of alpha=1 works for an example function I made (I say "think" because it seems to be pretty intermittent). Here is the example function, with the noiseless and noised version:

You can see that the noise version has a lot, but the ideal function does have a well enough defined maximum that it should be findable if it searches intelligently. And it tends to find it (repeating it for the same number of iterations):

it also sometimes still fails, like this:

But, this is a really simple contrived example function, with a known range and noise level. My real function, I'm not exactly sure what the range will be.
It feels like I've basically traded randomly searching for one hyperparameter with randomly searching for another.
If 1e-10 is "no noise" and 1.0 seems to work, that's...quite a range. Is there some strategy for figuring out what alpha value to choose? 
 A: According to scipy documentation, alpha is the standard deviation of the sampling noise at your data point. This is useful for applications such as ones in experimental physics where you have instrument errors, in which case you would just calculate your expected noise and use it as alpha. If you are tuning hyperparameters for machine learning to optimize for some variable, however, you would have to estimate this. Monte Carlo may be a good way for this estimation.
Kappa controls how much you value uncertainty. During the optimization, if you have found a good region but still have some parameter space unexplored, a larger Kappa means that you will want to explore the unknown parameter more and a smaller Kappa means that you want to try out points in this local area that you expect to have good values first. This parameter is slightly more tricky to tune. In theory, it does not depend too much on how volatile your function is and people seem to just use fixed values for this such as e=2.71828...
Example of kappa people chose: Bayes Optimization in R
