12
$\begingroup$

The Gaussian process has been widely used, especially in emulation. It is known that the computational demand is high ($0(n^3)$).

  1. What makes them popular?
  2. What are their main and hidden advantages?
  3. Why are they used instead of parametric models (by parametric model I mean typical linear regression in which different parametric forms can be used to describe the input vs output trend; e.g., qaudratic)?

I would really appreciate a technical answer explaning the inherent properties that make the Gaussian process unique and advantageous

$\endgroup$
8
  • $\begingroup$ Can you clarify what do you mean by parametric models? $\endgroup$ Apr 13 '16 at 20:56
  • $\begingroup$ @Alexey I have clarified what I mean by paramtric model above . Thank you $\endgroup$
    – Wis
    Apr 13 '16 at 20:58
  • $\begingroup$ From what I assume about parametric models you need to specify model by hand for each problem. This is not always possible, as true nature is not always known. Moreover, there can be difficulties with fitting of these models, while for Gaussian processes parameters estimation works well almost every time. $\endgroup$ Apr 13 '16 at 20:59
  • 4
    $\begingroup$ Splines and linear regression are equivalent to Gaussian processes regression with proper covariance function selected. But Gaussian processes provide a convenient probabilistic framework well suited for many tasks. $\endgroup$ Apr 13 '16 at 21:02
  • 1
    $\begingroup$ When would you not use Gaussian Process? $\endgroup$
    – Alby
    Aug 28 '17 at 19:58
6
$\begingroup$

The main advantages are from the engineering point of view (as @Alexey mentioned). In the widely used Kriging procedure you can interpret your own "space" by providing a "correlation" (or covariance) model (typically called the variogram ellipsoid) for relations depending on distance and orientation.

There is nothing that prevents other methodologies to have the same features, it just happened that the way kriging was first conceptualized had a friendly approach to people that were not statisticians.

Nowadays with the rise of geostatistics based stochastic methodologies, like Sequential Gaussian Simulation among others, these procedures are getting used in sectors where it is important to define the uncertainty space (which can take thousands to millions of dimensions). Again, from the engineering point of view, geostatistics based algorithms are very easy to include in genetic programming. As so when you have inverse problems you need to able to test multiple scenarios and test their adaptability to your optimization function.

Let's leave the pure argumentation for a moment a state the facts for a modern real example of this use. You can either sample underground samples directly (hard-data) or make a seismic map of the subsurface (soft-data).

In hard data you can measure a property (let's say acoustic impedance) directly without(ish) error. The problem is that this is scarce (and expensive). On the other hand you have the seismic mapping which is literally a volume, pixel-wise, map of the subsurface but does not give you acoustic impedance. For simplicity purposes let's say it gives you the ratio between two values of acoustic impedance (top and bottom). So a ratio of 0.5 could be a division of 1000/2000 or 10 000/ 20 000. It's a multiple solution space and several combinations will do but only one accurately represents reality. How do you solve this?

The way seismic inversion works (the stochastic procedures) is by producing plausible (and this is another story all together) scenarios of acoustic impedance (or other properties), transform those scenarios into a synthetic seismic (like the ratio in the previous example) and compare the synthetic seismic against the real one (correlation). The best scenarios will be used to produce even more scenarios, converging into a solution (this is not as easy as it seems).

Taking this into account and speaking from the point of view of usability I would answer your questions the following way:

1) What makes them popular is usability, flexibility in implementation, a good number of research centers and institutions that keep making newer and more adaptable gaussian based procedures for several different fields (particularly in geosciences, GIS included).

2) The main advantages are, as mentioned before, usability and flexibility from my point of view. If it's easy to manipulate and easy to use you just do it. There are no particular features in gaussian processes that are not reproducible in other methodologies (statistics or otherwise).

3) They are used when you need to include more information into your model than just the data (information such has space wise relations, statistical distributions, and so on...). I can assure that if you have lot's of data with an isotropic behavior using kriging is a waste of time. You can get the same results using any other method that by requiring less information, its faster to run.

$\endgroup$
7
  • $\begingroup$ And when is another model a better choice? $\endgroup$
    – Ben
    Sep 2 '19 at 14:28
  • 1
    $\begingroup$ @Ben It will always depend on the case study. Kriging, or Kriging based methods, have a high computation cost (so, not fast). For example, modern 4k (or more) TVs use interpolation methods to try and improve content that was made for smaller resolutions. This implies that it needs to perform this operation fast and without user intervention (which a covariance model would require). If I were to solve this particular problem I would avoid Kriging altogether. Moreover some phenomena is pattern based, or has a discrete variable, or can be reduced to formula (FEM, for example), etc... $\endgroup$
    – armatita
    Sep 2 '19 at 20:37
  • $\begingroup$ And when speed isn't important? $\endgroup$
    – Ben
    Sep 3 '19 at 9:31
  • $\begingroup$ @Ben Speed is less important if your result does not need to be immediate. Subsurface modeling, weather prediction, and bunch of operations within the GIS sciences are just a few examples. Another is the one presented in the answer (seismic inversion). $\endgroup$
    – armatita
    Sep 3 '19 at 11:08
  • $\begingroup$ Sorry, didn't get that. Neither computational nor result speed matters, what are disadvantages of a GP? Or in other words: Shouldn't it be used much more often? $\endgroup$
    – Ben
    Sep 3 '19 at 11:33
7
$\begingroup$

For engineers it is important:

  • to have confidence intervals for predictions
  • to interpolate training data
  • to have smooth and nonlinear models
  • use obtained regression models for adaptive design of experiments and optimization

Gaussian processes meet all these requirements.

Moreover, often engineering and geostatistics data sets are not that big or have specific grid structure allowing fast inference.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your comment . It seems due to their bayesian interpretation gaussian process models can have good uncertainty quantification , however this is also possible in parametric regression . I am looking for a technical approach that can explain the set of statistical advantages $\endgroup$
    – Wis
    Apr 13 '16 at 20:54
3
$\begingroup$

The Advantages of Gaussian Model.

Gaussian PDF only depends on its 1st-order and 2nd-order moments. A wide-sense stationary Gaussian process is also a strict-sense stationary process and vice versa.

Gaussian PDFs can model the distribution of many processes including some important classes of signals and noise. The sum of many independent random processes has a Gaussian distribution (central limit theorem).

Non-Gaussian processes can be approximated by a weighted combination (i.e. a mixture) of a number of Gaussian pdfs of appropriate means and variances.

Optimal estimation methods based on Gaussian models often result in linear and mathematically tractable solutions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.