what is the difference between Bayesian optimization and kriging? Both methods use Gaussian process, and kriging uses the Best Linear Unbiased Predictor (BLUP) to predict the mean (this is not seen in Bayesian optimization?). At the bottom line, they also have covariance matrix, whose inverse has to be computed before moving out to the next sample point. As far as I understand,


*

*Bayesian optimization yields a posterior pdf with mean and variance

*kriging yields a predicted mean and MSE $\sigma^2$


Obviously they are different for some reasons. Why are they different?
 A: I debated answering this because I have not used kriging in probably twelve years.  They are also closely related.  It is somewhat of a bridge between the extremes.  Still, that kriging is the BLUP isn't an advantage intrinsically over Bayesian methods.  Bayesian methods are not unbiased, though they are also not biased.  Instead, they ignore the question of bias.
You could probably make an argument that Bayesian methods are intrinsically biased because you are only unbiased at exactly one point and that by generally ignoring bias the probability of hitting that point is zero unless you were trying to mimic an unbiased estimator.  
The problem of unbiased estimators, generally, is that they are never less risky than a Bayesian estimator and so Bayesian estimators can be far more accurate than a non-Bayesian estimator.  There are two sources of this difference.  The first is that when bias exists, it is caused by information from outside the sample.  The second is that non-Bayesian probability distributions are mini-max distributions and not true probability distributions.
In a variety of circumstances, there is no observable difference between the two methods, at all.  The one thing you are missing is that Bayesian prediction is usually through the Bayesian predictive distribution and not the posterior.  While the posterior is $\Pr(\theta|x_1\dots{x_n})$, the Bayesian predictive distribution is $\Pr(x_{n+1}|x_1\dots{x_n})$.  Notice there is no parameter in that mathematical statement.  You can still do point optimization over a predictive distribution instead of a posterior.  
The advantage of kriging over Bayesian methods is that you can feel assured that the solution will be unbiased.  The Bayesian solution has two advantages.  The first is that they are never less risky, but they are also "coherent."  A statistic is coherent if a bookie could not face a certain loss by gambling on a statistics's result.
That said, kriging is structurally closer to Bayesian methods than probably any other without giving up the unbiased guarantee.
There is one other difference that is trivial for users who actually know what kriging is, but isn't theoretically trivial.  Bayesian methods are always subjective, even when they mimic unbiased methods.  A creationist testing evolution would find an experiment unconvincing with a proper prior if their beliefs were extreme enough.  It is unlikely you will be a fundamentalist in some discipline if kriging is your other solution because if you have gone to all the effort to pick up that much skill and experience, you are unlikely to let your personal dispositions overwhelm the data.
A: I believe you mean Gaussian processes rather than Bayesian optimisation. Bayesian optimisation is the use of Gaussian processes for global optimisation. Essentially you use the mean and variance of your posterior Gaussian process to balance the exploration and exploitation trade off in global optimisation (i.e. You want to fin the highest local point but you don't want to fall into local extrema).
Gaussian processes have been around since the 60s as far as I'm aware and maybe even earlier than that. As such they have been used and modified in different fields. In geostatistics, which at one stage was dominated by the French research community, they became 'kriging' and many scaling approximations and kernels were derived specifically for geo-spatial low dimensional data. 
In statistics and later machine learning they remained being referred to as Gaussian processes. Unfortunately this occasionally lead to people reinventing the wheel when it came to approximate approaches and theoretical analysis.
Now it's not uncommon that people use the terms interchangeably. For example Tom Nickson created a kronecker product sparse variation along approach Gaussian Processes which he called Blitz-Kriging.
