# Problem with prior mean in MOE (Bayesian optimization)

I am playing with MOE package (yelp.github.io/MOE) - I try to optimize some function of one variable, adding one point for sample at a time. Here is the intermediate chart I got:

Blue line is the actual function I optimize, red line - posterior mean from Gaussian Process. Green lines - standard deviations.

In my opinion, there is a huge problem here, because according to Bayesian optimization approach (as I understand it), samples are supposed to be taken from GP with prior constant mean (near zero).

And here I see posterior mean far from zero. It is very unlikely that this mean was taken from prior GP with near-zero constant mean. Changing covariance parameters doesn't help, I didn't see any effect at all. Noise is zero.

Any ideas on this (strange) behavior?

In my opinion, there is a huge problem here, because according to Bayesian optimization approach (as I understand it), samples are supposed to be taken from GP with prior constant mean (near zero).

And here I see posterior mean far from zero. It is very unlikely that this mean was taken from prior GP with near-zero constant mean. Changing covariance parameters doesn't help, I didn't see any effect at all. Noise is zero.

My understanding is that either MOE has a constant prior mean or the users supplies a constant prior mean (that is, $$\mu\sim\mathcal{N}(c,\sigma^2),$$ for some constant $$c$$, as opposed to the prior on the mean $$\mu$$ being some sort of function such as $$\mu\sim\mathcal{N}(X\beta,\sigma^2)$$ for some predictors $$X$$ and coefficients $$\beta$$). But this is just the prior. Simultaneously estimating the posterior GP parameters and the posterior mean parameters will shift both of these, so there's no reason to believe that the posterior will lie anywhere near zero.

What's probably happened here is that the model has found some solution to estimating the GP parameters such that the posterior mean is far from zero (say, $$\mu=-10$$), and the effect of the covariance parameters is to bring the posterior predictions in line with your observed data points. I'm not sure what estimation procedure you used for this model, but there's no guarantee that the likelihood of a GP mdoel does not suffer from multiple local maxima; that is, it's possible that there is an equally good (or even better!) model in terms of its likelihood, so this solution is just one among many (Rasmussen and Williams, Guassian Processes for Machine Learning, page 115). One of these alternative models may have a different combination of mean and covariance parameters such that the posterior mean is closer to all of your data points.

One solution to this indeterminacy is Bayeisan GP inference: taking a posterior distribution over all GP parameter estimates, the analyst effectively takes a weighted average over alternative GP models, with the weighting corresponding to its posterior probability. More information can be found in Gelman, Bayesian Data Analysis, 3rd ed., which has an entire chapter on Bayesian GP inference.

While it's the GP model is so quite far from your actual model in that middle region, I believe that's almost entirely the effect of not having any sample values in that interval. When you sample a point in there and re-estimate the GP parameters, your posterior predictive function should stick much closer to your function points. The general point here is that having more data will permit better model predictions, and that those predictions will tend to be better the closer they are to sample points!

I'm not sure what method you've used to generate these sample points, but I believe MOE implements EGO (Jones, "Efficient Global Optimization of Expensive Black-Box Functions," Journal of Global Optimization, 1998) for parameter estimation, which is highly regarded as a global optimization algorithm. If your goal is to optimize an arbitrary function, you could do much worse than using EGO.

Finally figured it out. user777, thanks for the insightful info!

I want to put it very clearly in case anyone will face the same difficulties. Two main points:

1. MOE establish prior mean as ZERO always. Even if you have initially sampled points where function values are in interval [-100, -101], anyway prior mean would be ZERO.
2. Mean, Var are calculated separately from GP itself. So if you optimized hyperparameters of GP, that doesn't mean that if you draw a mean, it will fit well. Most probably, you will see some kind of mess (like the one on the picture), because this mean would be calculated using DEFAULT hyperparameters (covariance parameters), not optimized ones. Once you drag optimized parameters from GP and put them into calculation of mean, picture will be quite pretty.

=> Conclusions: 1. Extract hyperparameters from GP , use them in all related calculations. 2. Need preprocess functions: scaling and translations, in order it to be somewhere around zero with st. dev. around 1.