3
$\begingroup$

Both methods use Gaussian process, and kriging uses the Best Linear Unbiased Predictor (BLUP) to predict the mean (this is not seen in Bayesian optimization?). At the bottom line, they also have covariance matrix, whose inverse has to be computed before moving out to the next sample point. As far as I understand,

  • Bayesian optimization yields a posterior pdf with mean and variance
  • kriging yields a predicted mean and MSE $\sigma^2$

Obviously they are different for some reasons. Why are they different?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

I believe you mean Gaussian processes rather than Bayesian optimisation. Bayesian optimisation is the use of Gaussian processes for global optimisation. Essentially you use the mean and variance of your posterior Gaussian process to balance the exploration and exploitation trade off in global optimisation (i.e. You want to fin the highest local point but you don't want to fall into local extrema).

Gaussian processes have been around since the 60s as far as I'm aware and maybe even earlier than that. As such they have been used and modified in different fields. In geostatistics, which at one stage was dominated by the French research community, they became 'kriging' and many scaling approximations and kernels were derived specifically for geo-spatial low dimensional data.

In statistics and later machine learning they remained being referred to as Gaussian processes. Unfortunately this occasionally lead to people reinventing the wheel when it came to approximate approaches and theoretical analysis.

Now it's not uncommon that people use the terms interchangeably. For example Tom Nickson created a kronecker product sparse variation along approach Gaussian Processes which he called Blitz-Kriging.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ thank you for bringing in the interesting rich history. it quite bothers me that people tend to use the terms interchangeably, because technically to me they seem to be different(?). while both uses Gaussian process, kriging uses BLUP as predictors and to be honest, i don't see GP uses BLUP. GP uses conditional probability on the observation datatset, as $\text{Pr}(x_{n+1} | x_1, \cdots. x_n )$, as @Dave Harris points out. so in that sense, fundamentally i think they are different, unless there is a connection between kriging predictor and GP predictor? $\endgroup$
    – kensaii
    Commented Feb 24, 2017 at 15:14
  • $\begingroup$ thanks for the introduction to blitzkriging. it was something new :) $\endgroup$
    – kensaii
    Commented Feb 24, 2017 at 15:16
5
$\begingroup$

I debated answering this because I have not used kriging in probably twelve years. They are also closely related. It is somewhat of a bridge between the extremes. Still, that kriging is the BLUP isn't an advantage intrinsically over Bayesian methods. Bayesian methods are not unbiased, though they are also not biased. Instead, they ignore the question of bias.

You could probably make an argument that Bayesian methods are intrinsically biased because you are only unbiased at exactly one point and that by generally ignoring bias the probability of hitting that point is zero unless you were trying to mimic an unbiased estimator.

The problem of unbiased estimators, generally, is that they are never less risky than a Bayesian estimator and so Bayesian estimators can be far more accurate than a non-Bayesian estimator. There are two sources of this difference. The first is that when bias exists, it is caused by information from outside the sample. The second is that non-Bayesian probability distributions are mini-max distributions and not true probability distributions.

In a variety of circumstances, there is no observable difference between the two methods, at all. The one thing you are missing is that Bayesian prediction is usually through the Bayesian predictive distribution and not the posterior. While the posterior is $\Pr(\theta|x_1\dots{x_n})$, the Bayesian predictive distribution is $\Pr(x_{n+1}|x_1\dots{x_n})$. Notice there is no parameter in that mathematical statement. You can still do point optimization over a predictive distribution instead of a posterior.

The advantage of kriging over Bayesian methods is that you can feel assured that the solution will be unbiased. The Bayesian solution has two advantages. The first is that they are never less risky, but they are also "coherent." A statistic is coherent if a bookie could not face a certain loss by gambling on a statistics's result.

That said, kriging is structurally closer to Bayesian methods than probably any other without giving up the unbiased guarantee.

There is one other difference that is trivial for users who actually know what kriging is, but isn't theoretically trivial. Bayesian methods are always subjective, even when they mimic unbiased methods. A creationist testing evolution would find an experiment unconvincing with a proper prior if their beliefs were extreme enough. It is unlikely you will be a fundamentalist in some discipline if kriging is your other solution because if you have gone to all the effort to pick up that much skill and experience, you are unlikely to let your personal dispositions overwhelm the data.

Improve this answer
$\endgroup$
5
  • $\begingroup$ that was a great answer! just a personal question: which one do you prefer and why? thanks again! $\endgroup$
    – kensaii
    Commented Feb 23, 2017 at 5:25
  • $\begingroup$ plus, don't you have some hyperparameters regardless of Bayesian optimization or kriging? i.e. the parameters characterizing correlation models? I know these hyperparameters have to optimized within, subject to some objective functions, but would that make the Bayesian prediction distribution becomes $\text{Pr}(x_{n+1}| x_1,\cdots,x_n,\theta)$, rather than $\text{Pr}(x_{n+1}| x_1,\cdots,x_n)$? $\endgroup$
    – kensaii
    Commented Feb 23, 2017 at 5:41
  • 1
    $\begingroup$ I don't have a preference, rather, I tend to try to match the properties of my problem to the method. If my concern is "truth," in the yes/no sense, then I would prefer kriging. If I had a null I needed to falsify, then kriging wins. On the other hand, if I need to "find" something then I would use the Bayesian method. Even though they will likely come out the same or close in value, I let purpose decide. I answer "what am I doing and why," rather than "which is best?" That begs "best for what," which takes me back to my first question. $\endgroup$
    – Dave Harris
    Commented Feb 23, 2017 at 6:09
  • $\begingroup$ @kensaii hyperparameters do not change the nature of the Bayesian prediction. The parameters and any hyperparameters would have been marginalized out leaving only the data to condition the prediction on. To use the language of a mathematician I know, it creates a result without "the crutch of parameters." $\endgroup$
    – Dave Harris
    Commented Feb 23, 2017 at 6:24
  • $\begingroup$ I think you are not correct about the emphasis on parameter estimation over predictive performance - it simply depends what field you are in. In statistics parameter estimation can be important as it tells you about the underlying system where as in machine learning we care about predictive performance. However in both cases these are referred to as Gaussian processes (at least at the university where I research). It appears to be the geospatial stats guys who refer to and work on kriging. However, most of my colleagues and I are very Bayesian so maybe we are biased ourselves haha! ;) $\endgroup$
    – j__
    Commented Feb 23, 2017 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.