# How to select noninformative priors on hyperparameters in gaussian process regression

I want to do a full Bayesian analysis of a Gaussian process regression problem: $$x\sim{GP[\mu,K(t_i,t_j)]},$$ where $\mu$ and $K$ are mean and covariance functions and $$K(t_i,t_j)=\sigma_f^2\exp\left(-\frac{(t_i-t_j)^2}{2l^2}\right)+\sigma_n^2\delta(t_i,t_j),$$ where $\sigma_f^2$,$l$, and $\sigma_n^2$ are parameters of the covariance function. It seems like I can select a noninformative prior density on $\mu$ to be equal to $1$. How do I select noninformative hyperpriors for $\sigma_f^2$,$l$, and $\sigma_n^2$? Any reference will be very helpful.

There is no such thing as a truly noninformative prior--even the prior you've selected for the mean is not uniform under a reparameterisation.

Having said that, you could use proper uniform priors on the covariance and noise parameters, for example with a lower limit of 0 and an upper limit of $U$. Then you could vary $U$ to ensure that the arbitrary upper limit is not unduly affecting your results.

However, I would generally advocate using an informative prior... but that is another answer.

• Thanks for the suggestion! I have one more question though. Can we say that noninformative prior means the prior with minimum information? If so, can we find out an analytical expression for noninformative prior in this case similar to Jeffreys prior? Nov 17, 2017 at 20:36
• You are alluding to maximum entropy priors. Even in this case you have to input some information (the expected value conditions). Nov 17, 2017 at 21:01
• Can we differentiate between Maximum entropy and noninformative prior? Nov 17, 2017 at 21:21
• Noninformative does not really exist. There are various approaches that try to be "objective". I suggest you take a look at other questions to help answer this or ask a new question. Nov 17, 2017 at 21:24
• > There is no such thing as a truly noninformative prior--even the prior you've selected for the mean is not uniform under a reparameterisation. -- I don't know what you mean here. To look for non-informative priors, you look for parameterizations that make the likelihood function data translated and then apply a uniform prior on such a parameterization. See Ch 2 of Bayesian Inference in Statistical Analysis. Such parameterization may not always exits, but for some problems they do. Feb 6 at 0:46

Ren et al. derived reference priors (i.e. approximately noninformative priors) for Gaussian processes of this form in their paper Objective Bayesian Analysis for a Spatial Model with Nugget Effects

Using the notation from their paper, let $$\mathbf{y}$$ denote the observations, $$\mathbf{s}_1, \ldots, \mathbf{s}_n$$ denote the sampling locations, and $$\mathbf{X} = \left(\mathbf{x}(\mathbf{s}_1), \ldots, \mathbf{x}(s_n)\right)'$$ denote the $$n\times p$$ design matrix of a linear mean function.

Then $$\mathbf{y} \sim N_n(\mathbf{X}\mathbf{\beta}, \delta_1 \mathbf{G}), \quad\mathbf{G} = \eta\mathbf{I}_n + \mathbf{K}(\theta)$$ where $$\delta_1$$ is equivalent to $$\sigma_f^2$$ in your notation, $$\eta$$ is the noise-to-signal-ratio $$\sigma_n^2/\sigma_f^2$$, and $$\theta$$ represents the hyperparameters of the covariance function.

Then the reference prior $$\pi^R_*$$ (equation 23) from their paper is given by $$\pi^R_*(\theta, \eta, \delta_1, \beta) \propto \frac{1}{\delta_1} \left(\det\mathbf{\Sigma}_*(\theta, \eta)\right)^{1/2}$$ where $$\mathbf{\Sigma}_*(\theta, \eta) = \frac{1}{2} \begin{pmatrix} \mathrm{tr}\left(\mathbf{R}_G \frac{\partial}{\partial \theta} \mathbf{K}(\theta)\right)^2 & \mathrm{tr}\left(\mathbf{R}_G^2 \frac{\partial}{\partial \theta} \mathbf{K}(\theta)\right) & \mathrm{tr}\left(\mathbf{R}_G \frac{\partial}{\partial \theta} \mathbf{K}(\theta)\right) \\ * & \mathrm{tr}\left(\mathbf{R}_G^2\right) & \mathrm{tr}\left(\mathbf{R}_G\right) \\ * & * & n - p \\ \end{pmatrix}$$ and $$\mathbf{R}_G = \mathbf{G}^{-1} - \mathbf{G}^{-1} \mathbf{X} \left( \mathbf{X}'\mathbf{G}^{-1}\mathbf{X} \right)^{-1} \mathbf{X}' \mathbf{G}^{-1}$$

2. Noninformative priors in the simplified case where $$\sigma_n=0$$: Berger et al. Objective Bayesian Analysis of Spatially Correlated Data