7 votes

Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

You are correct. The Gaussian process is a distribution over functions. As with other Bayesian methods, you start with a prior and combine it with data (observed outcome) through likelihood to get a ...
user avatar
  • 115k
4 votes
Accepted

Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

In addition to Tim's answer, here are some slight nitpicks/clarifications which might assist your intuitive understanding/prevent possible confusion in the future: We start with a a-prior function, ...
user avatar
  • 713
4 votes

Marginalization over the nuisance variable

Though, there were great answers, specially from @gunes. The most generic case where there is no independence or conditional independence assumption, marginalising $\mathbf{f}$, over $\mathbf{u}$ ...
user avatar
  • 1,927
4 votes

Marginalization over the nuisance variable

As mentioned in the comments, the first multiplicand should be $p(y|u)$ because it's originally $p(y|f,u)$ and it's stated that $y$ and $f$ are conditionally independent given $u$. For the integral, ...
user avatar
  • 52.1k
4 votes

Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods

First of all, not only neural networks are universal approximators. There is nothing special about them, they just proved to work quite well for a class of problems. Kernel based methods generally don'...
user avatar
  • 115k
3 votes
Accepted

Marginalization over the nuisance variable

If you have the joint density $P(y,f,u)$ and you know that $f\perp y | u$, then you can rewrite the joint as $$P(y,f,u)=P(y,f|u)P(u)=P(y|u)P(f|u)P(u) \ \ (1)$$ Then for calculating to the marginal $P(...
user avatar
  • 1,681
3 votes
Accepted

Multi-kernel Gaussian Process model?

I am not sure if I understand you correctly. Yes, you can combine several kernels by adding and multiplying (cf. section 4.2.4 in Rasmussen/Williams, http://www.gaussianprocess.org/gpml/chapters/RW....
user avatar
  • 267
3 votes

How to improve Gaussian Process Regression fit: reducing oscillations and narrowing confidence interval

I’m not sure what is problem you’re trying to solve here. The prediction intervals are a feature, not a bug. The Gaussian process is a nonparametric, data-based model. The intervals are thin for ...
user avatar
  • 115k
3 votes

Why Gaussian process has marginalisation/consistency property?

A Gaussian process $\{X(t)\colon t \in \mathbb T\}$ is not defined as just a collection of Gaussian random variables; there is also the requirement that for every $n \geq 1$, every finite collection $...
user avatar
3 votes
Accepted

How to derive an inverse of Gaussian Kernel

This would be hard, at least for these commonly used kernels like Matern, squared exponential etc. Suppose that you can obtain an analytical solution of your $K^{-1}(x_i, x_i)$. Then GP regression ...
user avatar
  • 375
3 votes

Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods

Your question is more theoretically founded, but goes in the same direction as this one. You might want to check the answers there (disclaimer: one of them is mine). In order to answer your question ...
user avatar
  • 6,353
3 votes

Looking for a Picture that shows Statistical Models "Learning" from Data

It's easy to make your own. I'll do it in R. ...
user avatar
  • 32.1k
2 votes

Why Gaussian process has marginalisation/consistency property?

It is actually a good question which shows a subtlety of the definition of a general(not necessarily Gaussian) stochastic process. And I hope it is not too late for you. In GPML, it says A stochastic ...
user avatar
2 votes

Hyperparameter tuning in Gaussian Process Regression

If solving the linear problem $K\pmb{\alpha} = \textbf{y}$ is too expensive for you at each step of your optimisation, you could resort to approximation techniques such as the Nystr$\ddot{o}$m method (...
user avatar
2 votes
Accepted

What is the entropy of a Gaussian Process?

For stochastic processes, the usual generalisation of entropy is the entropy rate. See here: https://jsri.srtc.ac.ir/article-1-24-en.pdf [in case the link is broken in the future, look up "The ...
user avatar
2 votes
Accepted

Why is the posterior of a neural network gaussian process equal to the posterior of a neural network in the limit of infinite width layers?

See section 3 of the paper Emulating computer models with step-discontinuous outputs using Gaussian processes or section 15.4.5 of "Machine Learning: a probabilistic perspective" by Kevin ...
user avatar
2 votes

Laplace Approximation of Gaussian Classifier step clarification

When you want to transform multiple additive terms into multiple multiplicative terms, you factor. We see $(K^{-1}+W)^{-1}$ on both sides of $$ \mathbf{f} - (K^{-1} + W)^{-1} K^{-1} \mathbf{f} = (K^{-...
user avatar
2 votes

Gaussian process - what am I doing wrong?

I think with a large N you are sampling densely from [0, 1]. Some data points in x_train are very close to each other, causing K nearly singular. Consequently np.linalg.inv(K) will give you unstable ...
user avatar
2 votes
Accepted

Assumption of Gaussian mixture model

The sentence on i.i.d. variables says that for all $i$ $$ X_i \overset{i.i.d.}\sim f $$ This means that $X_i$'s are independent and each of them follows the distribution $f$. In the case of Gaussian ...
user avatar
  • 115k
2 votes
Accepted

Gaussian process with polynomial covariance function integrating to 0

I've done the math again and I have not found anything wrong with it. Here is a simple python snippet that shows that samples actually are zero mean... ...
user avatar
2 votes

Gaussian Process Regression: Is it possible to determine significance of terms?

Yes, there are ways to detect/incorporate the relevance of variables. One approach is called "Automatic Relevance Detection" (see Section 5.1. of Rasmussen/Williams). Covariance functions ...
user avatar
  • 1,730
2 votes
Accepted

How to improve Gaussian Process Regression fit: reducing oscillations and narrowing confidence interval

For the over/undershoot issue I recommend trying the Matern kernel, it can be seen as a generalization of a RBF with an extra parameter that controls smoothness. This extra parameter very often helps ...
user avatar
2 votes

Partial plots for Gaussian Process: definition and computation

I was also looking for partial plots for GPs. The 'brute' method described by scikit is a generic method that works with any estimator. This could be eventually extended to ALE plots. Then you could ...
user avatar
  • 21
2 votes
Accepted

Kernel design for Gaussian processes with multiple inputs

What kind of kernel design is commonly used when we don't want to make the assumption that the input variables have an equal impact? What you're probably looking for is a anisotropic kernel. The most ...
user avatar
  • 713
2 votes

Why are samples from Gaussian Process prior so "smooth"?

The covariance function encodes prior beliefs about the nature of the function. It basically says how similar the output of the Gaussian process should be as a function of the input features. If you ...
user avatar
2 votes
Accepted

What is the difference between a non-zero nugget and a noise term in Kriging/GPR?

Random noise and nugget effect are indeed quite similar to some extent. The difference between the two appears when there are repeated observations (i.e., several observations at the same location), ...
user avatar
2 votes

Looking for a Picture that shows Statistical Models "Learning" from Data

I would put emphasis on "the process of a statistical model trying to "learn" this "true ideal" function". This is clearly what gradient descent does. Consider a simple ...
user avatar
  • 93
2 votes
Accepted

Why is the covariance function of a Gaussian process computed for the inputs?

Remember that $m(x_i)$ and $\kappa(x_i,x_j)$ are formally defined as : $$m(x_i) = \mathbb E[f(x_i)],\quad \kappa(x_i,x_j)=\mathbb E[(f(x_i)-m(x_i))(f(x_j)-m(x_j))]$$ So, as we can see from the ...
user avatar
2 votes
Accepted

Is it possible to update a gaussian process individually for each observation?

I think we can start with decomposing $b$ into two subsets $b_1$ and $b_2$. The resulting partitioned Gaussian is given by $$ \begin{pmatrix} y_a\\y_{b_1}\\y_{b_2}\end{pmatrix} \sim \mathcal{N}\left[\...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible