# Tag Info

### Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

You are correct. The Gaussian process is a distribution over functions. As with other Bayesian methods, you start with a prior and combine it with data (observed outcome) through likelihood to get a ...
• 115k
Accepted

### Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

In addition to Tim's answer, here are some slight nitpicks/clarifications which might assist your intuitive understanding/prevent possible confusion in the future: We start with a a-prior function, ...
• 713

### Marginalization over the nuisance variable

Though, there were great answers, specially from @gunes. The most generic case where there is no independence or conditional independence assumption, marginalising $\mathbf{f}$, over $\mathbf{u}$ ...
• 1,927

### Marginalization over the nuisance variable

As mentioned in the comments, the first multiplicand should be $p(y|u)$ because it's originally $p(y|f,u)$ and it's stated that $y$ and $f$ are conditionally independent given $u$. For the integral, ...
• 52.1k

### Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods

First of all, not only neural networks are universal approximators. There is nothing special about them, they just proved to work quite well for a class of problems. Kernel based methods generally don'...
• 115k
Accepted

• 41.9k
Accepted

### How to derive an inverse of Gaussian Kernel

This would be hard, at least for these commonly used kernels like Matern, squared exponential etc. Suppose that you can obtain an analytical solution of your $K^{-1}(x_i, x_i)$. Then GP regression ...
• 375

### Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods

Your question is more theoretically founded, but goes in the same direction as this one. You might want to check the answers there (disclaimer: one of them is mine). In order to answer your question ...
• 6,353

### Looking for a Picture that shows Statistical Models "Learning" from Data

It's easy to make your own. I'll do it in R. ...
• 32.1k

### Why Gaussian process has marginalisation/consistency property?

It is actually a good question which shows a subtlety of the definition of a general(not necessarily Gaussian) stochastic process. And I hope it is not too late for you. In GPML, it says A stochastic ...

### Hyperparameter tuning in Gaussian Process Regression

If solving the linear problem $K\pmb{\alpha} = \textbf{y}$ is too expensive for you at each step of your optimisation, you could resort to approximation techniques such as the Nystr$\ddot{o}$m method (...
Accepted

### What is the entropy of a Gaussian Process?

For stochastic processes, the usual generalisation of entropy is the entropy rate. See here: https://jsri.srtc.ac.ir/article-1-24-en.pdf [in case the link is broken in the future, look up "The ...
Accepted

### Why is the posterior of a neural network gaussian process equal to the posterior of a neural network in the limit of infinite width layers?

See section 3 of the paper Emulating computer models with step-discontinuous outputs using Gaussian processes or section 15.4.5 of "Machine Learning: a probabilistic perspective" by Kevin ...

When you want to transform multiple additive terms into multiple multiplicative terms, you factor. We see $(K^{-1}+W)^{-1}$ on both sides of $$\mathbf{f} - (K^{-1} + W)^{-1} K^{-1} \mathbf{f} = (K^{-... 2 votes ### Gaussian process - what am I doing wrong? I think with a large N you are sampling densely from [0, 1]. Some data points in x_train are very close to each other, causing K nearly singular. Consequently np.linalg.inv(K) will give you unstable ... • 21 2 votes Accepted ### Assumption of Gaussian mixture model The sentence on i.i.d. variables says that for all i$$ X_i \overset{i.i.d.}\sim f $$This means that X_i's are independent and each of them follows the distribution f. In the case of Gaussian ... • 115k 2 votes Accepted ### Gaussian process with polynomial covariance function integrating to 0 I've done the math again and I have not found anything wrong with it. Here is a simple python snippet that shows that samples actually are zero mean... ... 2 votes ### Gaussian Process Regression: Is it possible to determine significance of terms? Yes, there are ways to detect/incorporate the relevance of variables. One approach is called "Automatic Relevance Detection" (see Section 5.1. of Rasmussen/Williams). Covariance functions ... • 1,730 2 votes Accepted ### How to improve Gaussian Process Regression fit: reducing oscillations and narrowing confidence interval For the over/undershoot issue I recommend trying the Matern kernel, it can be seen as a generalization of a RBF with an extra parameter that controls smoothness. This extra parameter very often helps ... • 191 2 votes ### Partial plots for Gaussian Process: definition and computation I was also looking for partial plots for GPs. The 'brute' method described by scikit is a generic method that works with any estimator. This could be eventually extended to ALE plots. Then you could ... • 21 2 votes Accepted ### Kernel design for Gaussian processes with multiple inputs What kind of kernel design is commonly used when we don't want to make the assumption that the input variables have an equal impact? What you're probably looking for is a anisotropic kernel. The most ... • 713 2 votes ### Why are samples from Gaussian Process prior so "smooth"? The covariance function encodes prior beliefs about the nature of the function. It basically says how similar the output of the Gaussian process should be as a function of the input features. If you ... • 48.9k 2 votes Accepted ### What is the difference between a non-zero nugget and a noise term in Kriging/GPR? Random noise and nugget effect are indeed quite similar to some extent. The difference between the two appears when there are repeated observations (i.e., several observations at the same location), ... 2 votes ### Looking for a Picture that shows Statistical Models "Learning" from Data I would put emphasis on "the process of a statistical model trying to "learn" this "true ideal" function". This is clearly what gradient descent does. Consider a simple ... • 93 2 votes Accepted ### Why is the covariance function of a Gaussian process computed for the inputs? Remember that m(x_i) and \kappa(x_i,x_j) are formally defined as :$$m(x_i) = \mathbb E[f(x_i)],\quad \kappa(x_i,x_j)=\mathbb E[(f(x_i)-m(x_i))(f(x_j)-m(x_j))]$$So, as we can see from the ... • 718 2 votes Accepted ### Is it possible to update a gaussian process individually for each observation? I think we can start with decomposing b into two subsets b_1 and b_2. The resulting partitioned Gaussian is given by$$ \begin{pmatrix} y_a\\y_{b_1}\\y_{b_2}\end{pmatrix} \sim \mathcal{N}\left[\...
• 640

Only top scored, non community-wiki answers of a minimum length are eligible