Gaussian Process Regression: Is it possible to determine significance of terms? Is there within the framework of Gaussian Process Regression a possibility to determine significance of terms as is the case in multiple linear regression?
In a multiple linear regression model, for instance
$y = \Sigma_{i=1}^N \beta_i \cdot x_i$ I can look at the significance of term $x_K$ by computing the p-value of the hypothesis test $\beta_K = 0$ and if this p-value is high, then I can take $x_K$ out of the model. Is there something similar using Gaussian Process Regression??
A closely related issue is to check for interaction between terms: If I have the model $y = \Sigma_{i=1}^N \beta_i \cdot x_i + \Sigma_{i,j} \beta_{ij} \cdot x_i \cdot x_j$ then I can test if $\beta_{K,L} = 0$ to check if there is interaction between $x_K$ and $x_J$. Is there something similar using Gaussian Process Regression??
 A: Yes, there are ways to detect/incorporate the relevance of variables. One approach is called "Automatic Relevance Detection" (see Section 5.1. of Rasmussen/Williams). Covariance functions are parameterized by hyper-parameters. One class of hyperparameters for stationary kernels is the correlation length, a scale factor on each variable. For the squared exponential kernel on $\mathbb{R}^n$ these are the $\lambda_i$ in
$$ K(x,y) = \exp \left\{ - \frac 12\sum_{i=1}^n \frac{(x_i-y_i)^2}{\lambda_i}\right\}.$$
These hyperparameters are typically fit using maximum likelihood or cross validation and many implementations of GP regression offer this out of the box.
The idea behind this is easiest to understand in the extreme case that your process does not depend on a certain variable $i$. In this case values of the process will be highly correlated (actually the same) over large distances in this direction of input space or in other words $\lambda_i\rightarrow\infty.$
In addition to this somewhat automatic process you have always the option to build up your covariance functions step by step. I.e. start with main effects only then (higher) interactions etc... and compare the fits. A systematic approach along these lines is given by the ANOVA decomposition (or ANOVA kernels). See this paper for example.
